containerized overcloud - neutron-openvswitchagent fails to start

[Rabi Mishra]

When deploying multi node overcloud with containers, neutron-openvswitchagent does not start on compute hosts with traceback[3].

deploy command:

#openstack overcloud deploy --templates /tmp/tripleo-heat-templates -e /tmp/tripleo-heat-templates/environments/docker.yaml -e /home/stack/containers-default-parameters.yaml -r roles_data.yaml

However adding '-e docker-network.yaml' creates the external bridge and ovsagent starts fine. But doing an re-deploy on an existing overcloud with the above mentioned environment does not fix it.

It seems like by default[1] ovs agent expects the external bridge for provider networks.

Probably it makes sense to have the external bridge created by default like the controller[2]

[1] http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/neutron-ovs-agent.yaml#n43

[2] http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-resource-registry-puppet.j2.yaml#n40

[3]

[heat-admin@overcloud-novacompute-0 ~]$ tail -20f /var/log/containers/neutron/neutron-openvswitch-agent.log
2017-05-17 08:16:20.667 21275 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting...
2017-05-17 08:16:20.668 21275 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected
2017-05-17 08:16:21.094 21275 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000fe29856e114a
2017-05-17 08:16:21.101 21275 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] Mapping physical network datacentre to bridge br-ex
2017-05-17 08:16:21.102 21275 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] Bridge br-ex for physical network datacentre does not exist. Agent terminated!
2017-05-17 08:16:21.102 21275 ERROR ryu.lib.hub [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2167, in main
    agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 183, in __init__
    self.setup_physical_bridges(self.bridge_mappings)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1096, in setup_physical_bridges
    sys.exit(1)
SystemExit: 1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/465468

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/482539

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brent Eagles (<email address hidden>) on branch: master
Review: https://review.openstack.org/482539

[OpenStack Infra (hudson-openstack)]

Change abandoned by Rabi Mishra (<email address hidden>) on branch: master
Review: https://review.openstack.org/465468

[Ihar Hrachyshka (ihar-hrachyshka)]

This patch is of relevance here: https://review.openstack.org/#/c/493695/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/497722

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/497723

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/493695
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/497723
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/497722
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/493695
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=76f130d6e8f7434433b2602af9794f1e9c742e1f
Submitter: Jenkins
Branch: master

commit 76f130d6e8f7434433b2602af9794f1e9c742e1f
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497722
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=0e47375b2f06a1c59f0cc25c1f38bde4bf1d9631
Submitter: Jenkins
Branch: master

commit 0e47375b2f06a1c59f0cc25c1f38bde4bf1d9631
Author: Steve Baker <email address hidden>
Date: Fri Aug 25 15:28:46 2017 +1200

    Don't use docker-network.yaml env

    All this network config does is create br-ex, which is now done in
    docker/services/neutron-ovs-agent.yaml regardless of what network
    config is used.

    Change-Id: I3b4d68118ff26b4574e4d527b62cc95c38b1d565
    Related-Bug: #1691403
    Depends-On: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/499137

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/499137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0f2af352d15cb9c486414619a8342e727a2e207c
Submitter: Jenkins
Branch: stable/pike

commit 0f2af352d15cb9c486414619a8342e727a2e207c
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556
    (cherry picked from commit 76f130d6e8f7434433b2602af9794f1e9c742e1f)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/497723
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=17e414311f52fa3e040777c8a7f0067be4bea9bf
Submitter: Jenkins
Branch: master

commit 17e414311f52fa3e040777c8a7f0067be4bea9bf
Author: Steve Baker <email address hidden>
Date: Fri Aug 25 15:31:55 2017 +1200

    Don't use docker-network.yaml env

    All this network config does is create br-ex, which is now done in
    docker/services/neutron-ovs-agent.yaml regardless of what network
    config is used.

    Change-Id: I841e2da42a019e6d8341dae168501fbaf57855e6
    Related-Bug: #1691403
    Depends-On: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc2 release candidate.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.