ovs-vswitchd needs to be forced to reconfigure after adding protocols to bridges

Bug #1852221 reported by Frode Nordahl on 2019-11-12
30
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack neutron-openvswitch charm
Undecided
Unassigned
Ubuntu Cloud Archive
Undecided
Unassigned
Train
High
Unassigned
Ussuri
Undecided
Unassigned
kolla-ansible
Undecided
Unassigned
neutron
Undecided
Unassigned
openvswitch
New
Undecided
Unassigned
neutron (Ubuntu)
Undecided
Unassigned
Eoan
High
Unassigned
Focal
Undecided
Unassigned
openvswitch (Ubuntu)
Undecided
Unassigned

Bug Description

[Impact]
When the neutron native ovs driver creates bridges it will sometimes apply/modify the supported openflow protocols on that bridge. The OpenVswitch versions shipped with Train and Ussuri don't support this which results in OF protocol mismatches when neutron performs operations on that bridge. The patch we are backporting here ensures that all protocol versions are set on the bridge at the point on create/init.

[Test Case]
 * deploy Openstack Train
 * go to a compute host and do: sudo ovs-ofctl -O OpenFlow14 dump-flows br-int
 * ensure you do not see "negotiation failed" errors

[Regression Potential]
 * this patch is ensuring that newly created Neutron ovs bridges have OpenFlow 1.0, 1.3 and 1.4 set on them. Neutron already supports these so is not expected to have any change in behaviour. The patch will not impact bridges that already exist (so will not fix them either if they are affected).

--------------------------------------------------------------------------

As part of programming OpenvSwitch, Neutron will add to which protocols bridges support [0].

However, the Open vSwitch `ovs-vswitchd` process does not appear to always update its perspective of which protocol versions it should support for bridges:

# ovs-ofctl -O OpenFlow14 dump-flows br-int
2019-11-12T12:52:56Z|00001|vconn|WARN|unix:/var/run/openvswitch/br-int.mgmt: version negotiation failed (we support version 0x05, peer supports version 0x01)
ovs-ofctl: br-int: failed to connect to socket (Broken pipe)

# systemctl restart ovsdb-server
# ovs-ofctl -O OpenFlow14 dump-flows br-int
 cookie=0x84ead4b79da3289a, duration=1.576s, table=0, n_packets=0, n_bytes=0, priority=65535,vlan_tci=0x0fff/0x1fff actions=drop
 cookie=0x84ead4b79da3289a, duration=1.352s, table=0, n_packets=0, n_bytes=0, priority=5,in_port="int-br-ex",dl_dst=fa:16:3f:69:2e:c6 actions=goto_table:4
...
(Success)

The restart of the `ovsdb-server` process above will make `ovs-vswitchd` reassess its configuration.

0: https://github.com/openstack/neutron/blob/0fa7e74ebb386b178d36ae684ff04f03bdd6cb0d/neutron/agent/common/ovs_lib.py#L281

James Page (james-page) on 2019-11-19
Changed in charm-neutron-openvswitch:
status: New → Invalid

Hello:

I've seen this behavior in OVS 2.12. Previous versions work well (2.11 and 2.10).

I detected this problem using Neutron (master), OVS 2.12 and OVS firewall. The firewall flows are added to the DB using "--bundle". This requires OF14. Because this protocol is added after the bridge is created, it does not work.

But if the protocol is added in the same transaction of the creation [1], then the "--bundle" add-flows commands work.

Regards.

[1] https://github.com/openstack/neutron/blob/4051e0b19dc9f318c2e0dd7c60eaa2c46536ad03/neutron/agent/common/ovs_lib.py#L286-L302

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in openvswitch (Ubuntu):
status: New → Confirmed
Frode Nordahl (fnordahl) on 2020-05-22
summary: - ovsdb-server needs to be restarted after adding protocols to bridges
+ ovs-vswitchd needs to be forced to reconfigure after adding protocols to
+ bridges
description: updated
Edward Hope-Morley (hopem) wrote :
description: updated
Changed in openvswitch (Ubuntu Focal):
status: New → Fix Released

The attachment "lp1852221-eoan-train.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
no longer affects: openvswitch (Ubuntu Focal)
no longer affects: openvswitch (Ubuntu Eoan)
Changed in neutron (Ubuntu Focal):
status: New → Fix Released
no longer affects: openvswitch (Ubuntu Eoan)
no longer affects: openvswitch (Ubuntu Focal)
Dan Streetman (ddstreet) wrote :

unsubscribing ubuntu-sponsors as openstack team will handle uploading this

Changed in neutron (Ubuntu Eoan):
importance: Undecided → High
status: New → Triaged
Corey Bryant (corey.bryant) wrote :

Thanks Ed. A new version of neutron has been uploaded to the eoan unapproved queue with your debdiff applied.
https://launchpad.net/ubuntu/eoan/+queue?queue_state=1&queue_text=neutron

Reviewed: https://review.opendev.org/733674
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7d9742004643d97fe76e9d461ab24090f2f705fa
Submitter: Zuul
Branch: stable/train

commit 7d9742004643d97fe76e9d461ab24090f2f705fa
Author: Jakub Libosvar <email address hidden>
Date: Tue Jan 14 11:30:10 2020 +0000

    Set OpenFlow 1.0, 1.3 and 1.4 by default on bridges

    There is a bug in OVS 2.12 where it's impossible to change protocol on
    a bridge. This patch should be reverted once OVS is fixed. More
    information about the bug at [1].

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=1782834

    Related-Bug: #1852221

    Change-Id: I1ead1eee48a0c56193f20797ab35be36a0458270
    (cherry picked from commit 0643ab44d8204cde78fb7e8713fdd46dad0d87df)

tags: added: in-stable-train

Hello Frode, or anyone else affected,

Accepted neutron into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/neutron/2:15.0.2-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in neutron (Ubuntu Eoan):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-eoan
Corey Bryant (corey.bryant) wrote :

Hello Frode, or anyone else affected,

Accepted neutron into train-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:train-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-train-needed to verification-train-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-train-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-train-needed
Edward Hope-Morley (hopem) wrote :

eoan verified with [Test Case] and output is:

root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# apt-cache policy neutron-openvswitch-agent
neutron-openvswitch-agent:
  Installed: 2:15.0.2-0ubuntu1.2
  Candidate: 2:15.0.2-0ubuntu1.2
  Version table:
 *** 2:15.0.2-0ubuntu1.2 500
        500 http://archive.ubuntu.com/ubuntu eoan-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     2:15.0.2-0ubuntu1.1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
     2:15.0.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu eoan/main amd64 Packages
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# sudo ovs-ofctl -O OpenFlow14 dump-flows br-int| grep -v cookie
OFPST_FLOW reply (OF1.4) (xid=0x2):
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu# grep negotiation /var/log/neutron/neutron-*
root@juju-38af90-lp1852221-eoan-6:/home/ubuntu#

tags: added: verification-done-eoan
removed: verification-needed-eoan
Edward Hope-Morley (hopem) wrote :

bionic-train verified with [Test Case] and output is:

ubuntu@hopem-bastion:~/stsstack-bundles/openstack$ juju ssh nova-compute/0 sudo -s
root@juju-dc1c82-lp1852221-train-6:~# apt-cache policy neutron-openvswitch-agent
neutron-openvswitch-agent:
  Installed: 2:15.0.2-0ubuntu1.2~cloud0
  Candidate: 2:15.0.2-0ubuntu1.2~cloud0
  Version table:
 *** 2:15.0.2-0ubuntu1.2~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu bionic-proposed/train/main amd64 Packages
        100 /var/lib/dpkg/status
     2:12.1.0-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
     2:12.0.1-0ubuntu1 500
        500 http://nova.clouds.archive.ubuntu.com/ubuntu bionic/main amd64 Packages
root@juju-dc1c82-lp1852221-train-6:~# sudo ovs-ofctl -O OpenFlow14 dump-flows br-int| grep -v cookie
OFPST_FLOW reply (OF1.4) (xid=0x2):
root@juju-dc1c82-lp1852221-train-6:~# grep negotiation /var/log/neutron/neutron-*
root@juju-dc1c82-lp1852221-train-6:~#

tags: added: verification-done verification-train-done
removed: verification-needed verification-train-needed

The verification of the Stable Release Update for neutron has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Corey Bryant (corey.bryant) wrote :

This bug was fixed in the package neutron - 2:15.0.2-0ubuntu1.2~cloud0
---------------

 neutron (2:15.0.2-0ubuntu1.2~cloud0) bionic-train; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 neutron (2:15.0.2-0ubuntu1.2) eoan; urgency=medium
 .
   * d/p/set-openflow-protocols-by-default-on-bridges.patch Backport fix to resolve
     issues with neutron adding OF protocols to bridges post-create (LP: #1852221).

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in neutron (Ubuntu):
status: New → Confirmed
Corey Bryant (corey.bryant) wrote :

Eoan is EOL in July 2020

Changed in neutron (Ubuntu Eoan):
status: Fix Committed → Won't Fix
Changed in neutron (Ubuntu):
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.