"Bridge br-ex for physical network public does not exist. Agent terminated!" in tempest-slow job(s) on stable branches

[Matt Riedemann]

Seen in a stable/stein change here:

http://logs.openstack.org/43/665143/1/check/tempest-slow-py3/c36a4a8/compute1/logs/screen-neutron-agent.txt.gz?level=TRACE#_Jun_24_10_58_51_041785

Jun 24 10:58:50.296667 ubuntu-bionic-ovh-gra1-0008250201 neutron-openvswitch-agent[15936]: ERROR ovsdbapp.backend.ovs_idl.idlutils [-] Unable to open stream to tcp:127.0.0.1:6640 to retrieve schema: Connection refused

Jun 24 10:58:51.041785 ubuntu-bionic-ovh-gra1-0008250201 neutron-openvswitch-agent[15936]: ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [None req-c7de08ad-67af-4e9f-91aa-37bc0ec08deb None None] Bridge br-ex for physical network public does not exist. Agent terminated!

This seems to only show up in the tempest-slow* jobs on stable branches for some reason:

http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Bridge%20br-ex%20for%20physical%20network%20public%20does%20not%20exist.%20Agent%20terminated!%5C%22%20AND%20tags%3A%5C%22screen-neutron-agent.txt%5C%22&from=7d

It also appears that this started around June 21.

The tempest-slow job runs slow scenario tests that other jobs like tempest-full won't run, e.g.:

TestNetworkAdvancedServerOps.test_server_connectivity_cold_migration
TestNetworkAdvancedServerOps.test_server_connectivity_live_migration
TestNetworkAdvancedServerOps.test_server_connectivity_resize

[Matt Riedemann (mriedem)]

This devstack stable/stein change merged around June 21 and does not run the tempest-slow job:

https://review.opendev.org/#/c/662274/

[Matt Riedemann (mriedem)]

I'm not really seeing anything in nova, neutron or tempest around June 21 that looks related to a regression.

Maybe there was a new package update in the bionic repo on June 21 which is causing a problem for ovs?

[Matt Riedemann (mriedem)]

> Maybe there was a new package update in the bionic repo on June 21 which is causing a problem for ovs?

If that were the case, we should see it on master as well, and shouldn't see it on stable/rocky where were still using xenial nodes.

[Slawek Kaplonski (slaweq)]

I have no idea why it started failing around 21st of June but IMO it is related to using lib/neutron instead of lib/neutron-legacy on subsides.

Please look at Devstack log from master branch ("good" run): http://logs.openstack.org/35/667035/4/check/tempest-slow-py3/ec61a4e/compute1/logs/devstacklog.txt.gz#_2019-06-24_15_31_24_020 - br-ex is created by devstack there.
And in case of failed run which You pointed, bridge isn't created.

I "fixed" it in master branch some time ago: https://review.opendev.org/#/c/662480/ and now proposed backport to stein: https://review.opendev.org/#/c/667216/ and rocky: https://review.opendev.org/#/c/667218/

[Slawek Kaplonski (slaweq)]

Proposed fixes:

https://review.opendev.org/#/q/1b15176b0598f00c442c3b40985df30204dc6963

[Ghanshyam Mann (ghanshyammann)]

added the tempest-slow in devtsack gate - https://review.opendev.org/#/c/667272/1

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667216
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=7a6b3c8a97c8d5e423e2df1a734d073d30442dcf
Submitter: Zuul
Branch: stable/stein

commit 7a6b3c8a97c8d5e423e2df1a734d073d30442dcf
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 31 16:23:57 2019 +0200

    Use neutron-legacy on subnodes in devstack zuul job

    There are still some issues with lib/neutron thus neutron-legacy is used
    on controller node in multinode jobs and in single node jobs.
    But in "group-vars" in devstack job it was configured to use
    lib/neutron which can cause some problems in multinode jobs.

    So lets switch to neutron-legacy on subnodes also until lib/neutron
    will be ready to use everywhere.

    Closes-Bug: #1834079

    Change-Id: I0d7f9f2baaee2836a719f199939156bd4f53f778
    (cherry picked from commit 1b15176b0598f00c442c3b40985df30204dc6963)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/667218
Committed: https://git.openstack.org/cgit/openstack/devstack/commit/?id=9ffa9aa53b6b7996b082a67e35c283c3ead3bb37
Submitter: Zuul
Branch: stable/rocky

commit 9ffa9aa53b6b7996b082a67e35c283c3ead3bb37
Author: Slawek Kaplonski <email address hidden>
Date: Fri May 31 16:23:57 2019 +0200

    Use neutron-legacy on subnodes in devstack zuul job

    There are still some issues with lib/neutron thus neutron-legacy is used
    on controller node in multinode jobs and in single node jobs.
    But in "group-vars" in devstack job it was configured to use
    lib/neutron which can cause some problems in multinode jobs.

    So lets switch to neutron-legacy on subnodes also until lib/neutron
    will be ready to use everywhere.

    Closes-Bug: #1834079

    Change-Id: I0d7f9f2baaee2836a719f199939156bd4f53f778
    (cherry picked from commit 1b15176b0598f00c442c3b40985df30204dc6963)
    (cherry picked from commit 7a6b3c8a97c8d5e423e2df1a734d073d30442dcf)