ovs-vswitchd needs to be forced to reconfigure after adding protocols to bridges

[Frode Nordahl]

[Impact]
When the neutron native ovs driver creates bridges it will sometimes apply/modify the supported openflow protocols on that bridge. The OpenVswitch versions shipped with Train and Ussuri don't support this which results in OF protocol mismatches when neutron performs operations on that bridge. The patch we are backporting here ensures that all protocol versions are set on the bridge at the point on create/init.

[Test Case]
 * deploy Openstack Train
 * go to a compute host and do: sudo ovs-ofctl -O OpenFlow14 dump-flows br-int
 * ensure you do not see "negotiation failed" errors

[Regression Potential]
 * this patch is ensuring that newly created Neutron ovs bridges have OpenFlow 1.0, 1.3 and 1.4 set on them. Neutron already supports these so is not expected to have any change in behaviour. The patch will not impact bridges that already exist (so will not fix them either if they are affected).

--------------------------------------------------------------------------

As part of programming OpenvSwitch, Neutron will add to which protocols bridges support [0].

However, the Open vSwitch `ovs-vswitchd` process does not appear to always update its perspective of which protocol versions it should support for bridges:

# ovs-ofctl -O OpenFlow14 dump-flows br-int
2019-11-12T12:52:56Z|00001|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x05, peer supports version 0x01)
ovs-ofctl: br-int: failed to connect to socket (Broken pipe)

# systemctl restart ovsdb-server
# ovs-ofctl -O OpenFlow14 dump-flows br-int
 cookie=0x84ead4b79da3289a, duration=1.576s, table=0, n_packets=0, n_bytes=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
 cookie=0x84ead4b79da3289a, duration=1.352s, table=0, n_packets=0, n_bytes=0, priority=5,in_port="int-br-ex",dl_dst=fa:16:3f:69:2e:c6 actions=goto_table:4
...
(Success)

The restart of the `ovsdb-server` process above will make `ovs-vswitchd` reassess its configuration.

0: https://github.com/openstack/neutron/blob/0fa7e74ebb386b178d36ae684ff04f03bdd6cb0d/neutron/agent/common/ovs_lib.py#L281

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Hello:

I've seen this behavior in OVS 2.12. Previous versions work well (2.11 and 2.10).

I detected this problem using Neutron (master), OVS 2.12 and OVS firewall. The firewall flows are added to the DB using "--bundle". This requires OF14. Because this protocol is added after the bridge is created, it does not work.

But if the protocol is added in the same transaction of the creation [1], then the "--bundle" add-flows commands work.

Regards.

[1] https://github.com/openstack/neutron/blob/4051e0b19dc9f318c2e0dd7c60eaa2c46536ad03/neutron/agent/common/ovs_lib.py#L286-L302

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Rodolfo Alonso (rodolfo-alonso-hernandez)]

Related patch in Neutron: https://review.opendev.org/#/c/698715/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/716538

[Radosław Piliszek (yoctozepto)]

This has been reported to Kolla via IRC.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.opendev.org/726010

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "lp1852221-eoan-train.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Dan Streetman (ddstreet)]

unsubscribing ubuntu-sponsors as openstack team will handle uploading this

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/733674

[Corey Bryant (corey.bryant)]

Thanks Ed. A new version of neutron has been uploaded to the eoan unapproved queue with your debdiff applied.
https://launchpad.net/ubuntu/eoan/+queue?queue_state=1&queue_text=neutron

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/733674
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7d9742004643d97fe76e9d461ab24090f2f705fa
Submitter: Zuul
Branch: stable/train

commit 7d9742004643d97fe76e9d461ab24090f2f705fa
Author: Jakub Libosvar <email address hidden>
Date: Tue Jan 14 11:30:10 2020 +0000

    Set OpenFlow 1.0, 1.3 and 1.4 by default on bridges

    There is a bug in OVS 2.12 where it's impossible to change protocol on
    a bridge. This patch should be reverted once OVS is fixed. More
    information about the bug at [1].

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1782834

    Related-Bug: #1852221

    Change-Id: I1ead1eee48a0c56193f20797ab35be36a0458270
    (cherry picked from commit 0643ab44d8204cde78fb7e8713fdd46dad0d87df)

[Łukasz Zemczak (sil2100)]

Hello Frode, or anyone else affected,

Accepted neutron into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:15.0.2-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Corey Bryant (corey.bryant)]

Hello Frode, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Edward Hope-Morley (hopem)]

eoan verified with [Test Case] and output is:

root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# apt-cache policy neutron-openvswitch-agent
neutron-openvswitch-agent:
  Installed: 2:15.0.2-0ubuntu1.2
  Candidate: 2:15.0.2-0ubuntu1.2
  Version table:
 *** 2:15.0.2-0ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:15.0.2-0ubuntu1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
     2:15.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan/main amd64 Packages
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# sudo ovs-ofctl -O OpenFlow14 dump-flows br-int| grep -v cookie
OFPST_FLOW reply (OF1.4) (xid=0x2):
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# grep negotiation /var/log/neutron/neutron-*
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu#

[Edward Hope-Morley (hopem)]

bionic-train verified with [Test Case] and output is:

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju ssh nova-compute/0 sudo -s
root@juju-dc1c82-lp1852221-train-6:~# apt-cache policy neutron-openvswitch-agent
neutron-openvswitch-agent:
  Installed: 2:15.0.2-0ubuntu1.2~cloud0
  Candidate: 2:15.0.2-0ubuntu1.2~cloud0
  Version table:
 *** 2:15.0.2-0ubuntu1.2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root@juju-dc1c82-lp1852221-train-6:~# sudo ovs-ofctl -O OpenFlow14 dump-flows br-int| grep -v cookie
OFPST_FLOW reply (OF1.4) (xid=0x2):
root@juju-dc1c82-lp1852221-train-6:~# grep negotiation /var/log/neutron/neutron-*
root@juju-dc1c82-lp1852221-train-6:~#

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.2~cloud0
---------------

 neutron (2:15.0.2-0ubuntu1.2~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:15.0.2-0ubuntu1.2) eoan; urgency=medium
 .
   * d/p/set-openflow-protocols-by-default-on-bridges.patch Backport fix to resolve
     issues with neutron adding OF protocols to bridges post-create (LP: #1852221).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Corey Bryant (corey.bryant)]

Eoan is EOL in July 2020

[Launchpad Janitor (janitor)]

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.2

---------------
neutron (2:15.0.2-0ubuntu1.2) eoan; urgency=medium

  * d/p/set-openflow-protocols-by-default-on-bridges.patch Backport fix to resolve
    issues with neutron adding OF protocols to bridges post-create (LP: #1852221).

 -- Edward Hope-Morley <email address hidden> Sun, 31 May 2020 16:50:01 +0100

[OpenStack Infra (hudson-openstack)]

Change abandoned by Frode Nordahl (<email address hidden>) on branch: master
Review: https://review.opendev.org/716538
Reason: The upstream Neutron workaround for this issue makes the charm handling redundant.

[Xav Paice (xavpaice)]

Seeing this in Focal, openvswitch version 2.13.0-0ubuntu1

[Xav Paice (xavpaice)]

re-reading this, the issue I was seeing was that the protocol wasn't negotiated - did not need to restart ovs to get the 'good' test. Apologies for the noise, it does actually look like this is also fixed in 2.13.0-0ubuntu1 and possibly could be updated for openvswitch (Ubuntu).

[James Page (james-page)]

2.15.0 contains the fix for this issue - marking Fix Released.

[James Page (james-page)]

Also backported to >= 2.13.2 so resolved for Ubuntu 20.04 LTS and later

[James Page (james-page)]

Fixed in v2.12.2 so still impacts the Train UCA (2.12.1 currently).

Earlier releases should not have this issue as don't contain the commit that regressed this feature.

$ git tag --contains a0baa7dfa4fe

v2.12.0
v2.12.1
v2.12.2
v2.12.3
v2.13.0
v2.13.1
v2.13.2
v2.13.3
v2.13.4
v2.14.0
v2.14.1
v2.14.2
v2.15.0
v2.15.1

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/812162

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Rodolfo Alonso <email address hidden>" on branch: stable/xena
Review: https://review.opendev.org/c/openstack/neutron/+/812162