containerized overcloud - neutron-openvswitchagent fails to start

Bug #1691403 reported by Rabi Mishra on 2017-05-17
20
This bug affects 3 people
Affects Status Importance Assigned to Milestone
tripleo
Critical
Steve Baker

Bug Description

When deploying multi node overcloud with containers, neutron-openvswitchagent does not start on compute hosts with traceback[3].

deploy command:

#openstack overcloud deploy --templates /tmp/tripleo-heat-templates -e /tmp/tripleo-heat-templates/environments/docker.yaml -e /home/stack/containers-default-parameters.yaml -r roles_data.yaml

However adding '-e docker-network.yaml' creates the external bridge and ovsagent starts fine. But doing an re-deploy on an existing overcloud with the above mentioned environment does not fix it.

It seems like by default[1] ovs agent expects the external bridge for provider networks.

Probably it makes sense to have the external bridge created by default like the controller[2]

[1] http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/neutron-ovs-agent.yaml#n43

[2] http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/overcloud-resource-registry-puppet.j2.yaml#n40

[3]

[heat-admin@overcloud-novacompute-0 ~]$ tail -20f /var/log/containers/neutron/neutron-openvswitch-agent.log
2017-05-17 08:16:20.667 21275 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connecting...
2017-05-17 08:16:20.668 21275 INFO ovsdbapp.backend.ovs_idl.vlog [-] tcp:127.0.0.1:6640: connected
2017-05-17 08:16:21.094 21275 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int has datapath-ID 0000fe29856e114a
2017-05-17 08:16:21.101 21275 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] Mapping physical network datacentre to bridge br-ex
2017-05-17 08:16:21.102 21275 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] Bridge br-ex for physical network datacentre does not exist. Agent terminated!
2017-05-17 08:16:21.102 21275 ERROR ryu.lib.hub [req-7e5dfd8c-7792-4c9b-bcac-7969ae36712e - - - - -] hub: uncaught exception: Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ryu/lib/hub.py", line 54, in _launch
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/ovs_ryuapp.py", line 40, in agent_main_wrapper
    ovs_agent.main(bridge_classes)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 2167, in main
    agent = OVSNeutronAgent(bridge_classes, cfg.CONF)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 183, in __init__
    self.setup_physical_bridges(self.bridge_mappings)
  File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 153, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/openvswitch/agent/ovs_neutron_agent.py", line 1096, in setup_physical_bridges
    sys.exit(1)
SystemExit: 1

Rabi Mishra (rabi) on 2017-05-17
Changed in tripleo:
assignee: nobody → Rabi Mishra (rabi)

Fix proposed to branch: master
Review: https://review.openstack.org/465468

Changed in tripleo:
status: New → In Progress
tags: added: networking
Changed in tripleo:
importance: Undecided → Medium
milestone: none → pike-2
Changed in tripleo:
milestone: pike-2 → pike-3

Fix proposed to branch: master
Review: https://review.openstack.org/482539

Changed in tripleo:
assignee: Rabi Mishra (rabi) → Brent Eagles (beagles)
Changed in tripleo:
milestone: pike-3 → pike-rc1
Changed in tripleo:
milestone: pike-rc1 → queens-1

Change abandoned by Brent Eagles (<email address hidden>) on branch: master
Review: https://review.openstack.org/482539

Changed in tripleo:
importance: Medium → High
assignee: Brent Eagles (beagles) → Steve Baker (steve-stevebaker)
milestone: queens-1 → pike-rc1

Change abandoned by Rabi Mishra (<email address hidden>) on branch: master
Review: https://review.openstack.org/465468

Changed in tripleo:
assignee: Steve Baker (steve-stevebaker) → Ian Main (imain)
wes hayutin (weshayutin) on 2017-08-22
tags: added: alert ci
Changed in tripleo:
importance: High → Critical
Changed in tripleo:
assignee: Ian Main (imain) → Steve Baker (steve-stevebaker)

This patch is of relevance here: https://review.openstack.org/#/c/493695/

tags: added: containers
Changed in tripleo:
milestone: pike-rc1 → pike-rc2

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/493695
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/497723
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/497722
Reason: I need to purge the gate because TripleO CI gate has critical issues right now, I'll make this patch goes to the gate.

Reviewed: https://review.openstack.org/493695
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=76f130d6e8f7434433b2602af9794f1e9c742e1f
Submitter: Jenkins
Branch: master

commit 76f130d6e8f7434433b2602af9794f1e9c742e1f
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556

Changed in tripleo:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/497722
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart/commit/?id=0e47375b2f06a1c59f0cc25c1f38bde4bf1d9631
Submitter: Jenkins
Branch: master

commit 0e47375b2f06a1c59f0cc25c1f38bde4bf1d9631
Author: Steve Baker <email address hidden>
Date: Fri Aug 25 15:28:46 2017 +1200

    Don't use docker-network.yaml env

    All this network config does is create br-ex, which is now done in
    docker/services/neutron-ovs-agent.yaml regardless of what network
    config is used.

    Change-Id: I3b4d68118ff26b4574e4d527b62cc95c38b1d565
    Related-Bug: #1691403
    Depends-On: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb

Reviewed: https://review.openstack.org/499137
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=0f2af352d15cb9c486414619a8342e727a2e207c
Submitter: Jenkins
Branch: stable/pike

commit 0f2af352d15cb9c486414619a8342e727a2e207c
Author: Steve Baker <email address hidden>
Date: Tue Aug 15 10:26:24 2017 +1200

    container ovs-agent, ensure br-ex exists

    Currently the container neutron-ovs-agent is stuck in a restart loop
    in many environments because the bridge br-ex is missing.

    This bridge is created by running the puppet class
    neutron::agents::ml2::ovs but limiting that run to tag
    neutron::plugins::ovs::bridge.

    The hiera neutron::agents::ml2::ovs::bridge_mappings should already
    exists to create the bridge with the required settings.

    This change should ensure br-ex exists after step 3.

    Since br-ex is created regardless of the chosen network config,
    environments/docker-network.yaml is not longer required. It can be
    deleted once there are no more references to it in CI and
    documentation.

    Change-Id: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb
    Closes-Bug: #1699261
    Closes-Bug: #1691403
    Closes-Bug: #1689556
    (cherry picked from commit 76f130d6e8f7434433b2602af9794f1e9c742e1f)

tags: added: in-stable-pike

Reviewed: https://review.openstack.org/497723
Committed: https://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/commit/?id=17e414311f52fa3e040777c8a7f0067be4bea9bf
Submitter: Jenkins
Branch: master

commit 17e414311f52fa3e040777c8a7f0067be4bea9bf
Author: Steve Baker <email address hidden>
Date: Fri Aug 25 15:31:55 2017 +1200

    Don't use docker-network.yaml env

    All this network config does is create br-ex, which is now done in
    docker/services/neutron-ovs-agent.yaml regardless of what network
    config is used.

    Change-Id: I841e2da42a019e6d8341dae168501fbaf57855e6
    Related-Bug: #1691403
    Depends-On: Ie425148b0ad0f38e149c5fa0a97d98ec35d0a5bb

This issue was fixed in the openstack/tripleo-heat-templates 7.0.0.0rc2 release candidate.

This issue was fixed in the openstack/tripleo-heat-templates 8.0.0.0b1 development milestone.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers