'br-prv' ovs bridge is missing after reboot of compute node

[Ksenia Svechnikova]

ISO 9.0-28

Deploy cluster with active-backup bonding and Neutron VLAN

Steps to reproduce:
            1. Create cluster
            2. Add 3 nodes with controller role
            3. Add 1 node with compute role and 1 node with cinder role
            4. Setup bonding for all interfaces (including admin interface
               bonding)
            5. Run network verification
            6. Deploy the cluster
            7. Run network verification
            8. Run OSTF
            9. Save network configuration from slave nodes
            10. Reboot all environment nodes
            11. Verify that network configuration is the same after reboot

Expected result: Verification pass

Actual result:
On step 11 we can see in the neutron-openvswitch-agent.log:

Mar 9 00:56:30 err: 2016-03-09 00:56:30.068 6390 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-4107d9e2-c48a-42d4-98b6-41e3d2adce2e - - - - -] Bridge br-prv for physical network physnet2 does not exist. Agent terminated!

Link to the swarm: https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.bonding_ha/39/

This reproduced also on VLAN non-bonding env after reboot one compute node

[Atsuko Ito (yottatsa)]

This is not l23network issue. Looks like it's upstart issue. I worked around this bug by removing /etc/init/cloud-* and /etc/init/failsafe.conf.

[Atsuko Ito (yottatsa)]

I had the same issues year ago in Yandex, when we rebuilded OpenVSwitch and start using upstart instear rc.d.
I recommend to rebuild our custom package back to rc.d.

To check this, execute next commands and reboot the machine:
rm /etc/init/openvswitch-switch.conf
update-rc.d openvswitch-switch defaults

[Atsuko Ito (yottatsa)]

see also https://bugs.launchpad.net/fuel/+bug/1553733

[Atsuko Ito (yottatsa)]

1553733 was merged, please revalidate

[Ksenia Svechnikova (kdemina)]

The issue is still present. I've reproduced it on my venv with HugePages. Attach the snapshot:

[Ksenia Svechnikova (kdemina)]

Confirm at ISO 92

[Atsuko Ito (yottatsa)]

Could not reproduce swarm because of https://bugs.launchpad.net/fuel/+bug/1559025

[Atsuko Ito (yottatsa)]

Bug was hidden by https://bugs.launchpad.net/fuel/+bug/1548296

[Atsuko Ito (yottatsa)]

Waiting for tonight swarm

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Oleg Bondarev (obondarev)]

Bug https://bugs.launchpad.net/fuel/+bug/1564934 looks the same: after controller restart br-floating was not created so ovs agent failed to start:
 2016-04-01 00:15:53.545 21019 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-24b0cb67-4150-486c-9b66-a89cd9d58f93 - - - - -] Bridge br-floating for physical network physnet1 does not exist. Agent terminated!

[Aleksandr Didenko (adidenko)]

https://product-ci.infra.mirantis.net/view/9.0_swarm/job/9.0.system_test.ubuntu.bonding_ha/ is green for 3 days already.
I've run a test on fuel-9.0-191-2016-04-12_02-00-00.iso

1) Run 'bonding_ha' systest group (it's successful)
2) Revert-resume deploy_bonding_neutron_vlan
3) Reboot 2 controllers and 1 compute 200+ times (every 5 minutes during 16+ hours)

Then I've checked neutron logs on all nodes for "Bridge .* does not exist. Agent terminated" pattern. Zero matches.

I'll try to reproduce it with ha_neutron_destructive_vxlan test on the same ISO #191. But meanwhile I'm removing swarm blocker tag and marking this bug as incomplete, since it's not reproducable anymore.
If you're able to reproduce it on ISO newer than 9.0-191, then please post an update here with diagnostic snapshot. Thanks.

[Aleksei Stepanov (penguinolog)]

https://product-ci.infra.mirantis.net/job/9.0.system_test.ubuntu.network_templates/76/testReport/%28root%29/network_config_consistency_on_reboot/
ISO 9.0 193

[Aleksandr Didenko (adidenko)]

@Alexey, thanks for the snapshot. The test from snapshot uses custom network template where ALL bridges are OVS ones (provider=ovs). So it was much easier to reproduce and troubleshoot the issue.

OK, so it looks like we have 2 problems here:
1) Improper configuration of OVS bridges and ports in /etc/network/interfaces.d/ifcfg-* files (L23_stored_config in fuel-library)
2) Upstart/pre-if-up configuration for openvswitch-switch service.

Some details on each of those problems:
1) Let's take a look at stored config for br-ex - http://paste.openstack.org/show/494246/
If you run 'ifdown br-ex' and then 'ifup br-ex' this bridge will lose connection to physical network (ovs port enp0s4 will be missing from br-ex). It's happening because enp0s4 is not configured as OVSPort. Another problem is that both ports (enp0s4 and p_ff798dba-0) have "auto" enabled. So system tries to bring them up directly (not under bridge) when starts 'networking', which may fail, of course, since bridges are not yet created. Ports, connected to OVS bridge, should not have "auto" parameter (see examples here https://github.com/openvswitch/ovs/blob/master/debian/openvswitch-switch.README.Debian). Those ports will be brought up by /etc/network/if-pre-up.d/openvswitch script (see line #7 in this paste http://paste.openstack.org/show/494249/ ).
So we need to fix our manifests to configure OVS bridges and their ports like this: http://paste.openstack.org/show/494250/

2) The same /etc/network/if-pre-up.d/openvswitch script is the first one who brings openvswitch-switch service up (see http://paste.openstack.org/show/494251/). And it looks like the problem is related to ovsdb accessibility - it tries to configure OVS interfaces while ovsdb is not up yet (/var/run/openvswitch/db.sock is not accessible). After adding a simple wait for socket loop (see http://paste.openstack.org/show/494252/ ) the problem with missing OVS bridges was solved and I was able to find 'NO SOCKET' message in upstart networking log, so it obviously tried to execute ovs commands (which would fail without wait loop).

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: master
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/19782

[Aleksandr Didenko (adidenko)]

As for upstart part we can probably backport https://launchpad.net/ubuntu/+source/openvswitch/2.0.2-0ubuntu0.14.04.3

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19782
Submitter: Pkgs Jenkins <email address hidden>
Branch: master

Commit: a75ff1e6dc800246b65993f8128ccb3c046640ec
Author: Alexander Didenko <email address hidden>
Date: Tue Apr 19 09:20:05 2016

Fix race condition between networking and ovs

Networking upstart service checks if openvsiwtch-switch service
is running (and starts it if it's not running) via
debian/ifupdown.sh hook. But service may be in 'start/pre-start'
state in which it's loading kmod and not yet able to connect to
the database. Which leads to failures in interfaces configuration.

Backporting upstream fix for LP: #1314887

Change-Id: Ifc975caed9cd17c4d0ff5d9ddd00cad6e215f620
Partial-bug: #1555162

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 9.0
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/19889

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/19889
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0

Commit: 83e0a677518892162bd9370307c6daf660fbc306
Author: Alexander Didenko <email address hidden>
Date: Wed Apr 20 09:39:11 2016

Fix race condition between networking and ovs

Networking upstart service checks if openvsiwtch-switch service
is running (and starts it if it's not running) via
debian/ifupdown.sh hook. But service may be in 'start/pre-start'
state in which it's loading kmod and not yet able to connect to
the database. Which leads to failures in interfaces configuration.

Backporting upstream fix for LP: #1314887

Change-Id: Ifc975caed9cd17c4d0ff5d9ddd00cad6e215f620
Partial-bug: #1555162
(cherry picked from commit a75ff1e6dc800246b65993f8128ccb3c046640ec)

[Atsuko Ito (yottatsa)]

The problem with storing ovs ports as a linux ones because we're handling it strange way: even if provider is lnx, we still could add it in ovs bridge https://github.com/openstack/fuel-library/blob/master/deployment/puppet/l23network/lib/puppet/provider/l2_port/lnx.rb#L132

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/309052

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309052
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=645e01af3fc8dfdc4f33c0e9e8f43bdbcfd564c0
Submitter: Jenkins
Branch: master

commit 645e01af3fc8dfdc4f33c0e9e8f43bdbcfd564c0
Author: Vladimir Eremin <email address hidden>
Date: Thu Apr 21 17:53:22 2016 +0300

    Inherit provider for ports from bridges as default

    If Port provider is not specified and Bridge provider is OVS, we need to
    use OVS provider for Port instead of default one. Otherwise,
    L23_stored_config would choose wrong provider.

    Change-Id: I1213b70be19b6ce7324d69b1763d4bfd900fe3d9
    Closes-Bug: #1555162

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/309445

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/309445
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=81bf170400d01ccd18e4c5b2c47f0dbdaa9f7b0d
Submitter: Jenkins
Branch: stable/mitaka

commit 81bf170400d01ccd18e4c5b2c47f0dbdaa9f7b0d
Author: Vladimir Eremin <email address hidden>
Date: Thu Apr 21 17:53:22 2016 +0300

    Inherit provider for ports from bridges as default

    If Port provider is not specified and Bridge provider is OVS, we need to
    use OVS provider for Port instead of default one. Otherwise,
    L23_stored_config would choose wrong provider.

    Change-Id: I1213b70be19b6ce7324d69b1763d4bfd900fe3d9
    Closes-Bug: #1555162
    (cherry picked from commit 645e01af3fc8dfdc4f33c0e9e8f43bdbcfd564c0)

[Tatyanka (tatyana-leontovich)]

for mitaka verified on 285 iso

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Dmitry Teselkin <email address hidden> on branch: 9.0
Review: https://review.fuel-infra.org/25435
Reason: Merged in https://review.fuel-infra.org/#/q/topic:group/prod-7907

[Oleksiy Molchanov (omolchanov)]

https://review.fuel-infra.org/#/q/topic:group/prod-7907