[SRU] OVN 22.03.2 point release

[Frode Nordahl]

[Impact]
This release sports mostly bug-fixes and we would like to make sure all of our
supported customers have access to these improvements.
The update contains the following package updates:

   * ovn 22.03.2

[Test Case]

The following SRU process was followed:

https://wiki.ubuntu.com/OpenStack/StableReleaseUpdates

In order to avoid regression of existing consumers, the OpenStack team will
run their continuous integration test against the packages that are in
-proposed. A successful run of all available tests will be required before the
proposed packages can be let into -updates.

The OpenStack team will be in charge of attaching the output summary of the
executed tests. The OpenStack team members will not mark ‘verification-done’ until
this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned tests are attached to this bug.

[Frode Nordahl (fnordahl)]

Some background on the addition of flaky tests in the SRU package:
There are unfortunately a number of tests in the OVN testsuite that have timing issues and as a consequence appear as flaky when run under load in the Debian and Ubuntu build farms. The upstream is currently working through these issues, but it is a sizable undertaking, which will still take some time to fully fix.

We go through lengths to identify the underlying issue whenever a new test starts to fail, which can be observed in a recent bash regression which was caught by us [0].

The listed tests are passing when run on an unloaded system from the Debian/Ubuntu package sources, so there is no doubt that the functionality is intact in our composition of bits. For completeness I have included the output of a successful test run with the affected tests [1].

We regularly disable tests for the same reasons in Debian unstable with bug references [2][3][4][5][6][7], and I unfortunately did not go through the same rigor in tying these listings in the stable backport, for which I apologize.

0: https://bugs.launchpad.net/bugs/1997093
1: https://pastebin.ubuntu.com/p/R3RHfNcR7K/
2: https://salsa.debian.org/openstack-team/third-party/ovn/-/commit/923f0b1a3a4ade0a367a3be6db30f6b57acafe7e
3: https://salsa.debian.org/openstack-team/third-party/ovn/-/commit/cc2a217dc775cc60f704a13f85e6ec07933902e3
4: https://bugs.launchpad.net/bugs/2002406
5: https://bugs.launchpad.net/bugs/2002475
6: https://bugs.launchpad.net/bugs/2002476
7: https://bugs.launchpad.net/bugs/2002477

[Chris Halse Rogers (raof)]

An upload of ovn to jammy-proposed has been rejected from the upload queue for the following reason: "Unnecessary packaging changes in the disable-these-tests infrastructure. If you think they are necessary, please ping RAOF on #ubuntu-release and I can accept from the rejected queue.".

[Frode Nordahl (fnordahl)]

I respectfully disagree, the list of flaky tests was extended on the back of actual build failures found during pre-upload build testing on the Ubuntu build infrastructure through the use of an PPA (example in [8]).

As we've discussed on the kinetic OVN SRU bug 2003054 and as laid out in comment #1, the OVN testsuite unfortunately contain a number of flaky tests.

Jammy tracks the upstream LTS version of OVN and will keep getting regular updates through the course of the next year and not fixing this will make it harder to maintain the package.

8: https://launchpadlibrarian.net/646492517/buildlog_ubuntu-jammy-ppc64el.ovn_22.03.2-0ubuntu0.22.04.1.0_BUILDING.txt.gz

[Chris Halse Rogers (raof)]

Hello Frode, or anyone else affected,

Accepted ovn into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Frode Nordahl (fnordahl)]

Note that build and verification of this SRU can't proceed until the accompanying Open vSwitch SRU [0] gets accepted into the archive.

0: https://launchpadlibrarian.net/646504743/openvswitch_2.17.5-0ubuntu0.22.04.1_source.changes

[Ammad Ali (syedammad83)]

Hi Frode,

There is a bug identified in ovn 22.03.0-0ubuntu1~cloud0 FIP QoS not working properly (openstack neutron 20.2 yoga). Here is the bug description.

- create an instance.
- NAT a floating IP with instance.
- Apply FIP QoS 10Mbps.
- Then remove QoS from FIP.
- Apply FIP QoS 20Mbps.
- Remove 20Mbps QoS
- Apply FIP QoS 30Mbps
- The 30Mbps QoS will not applied instead the instance will get 20Mbps speed.
- Restart the ovn-controller on compute node.
- The instance will get 30Mbps speed.

The issue has been fixed in 22.03.1 and later minor release i think.

It would be great if the users can get ovs 2.17.5 and ovn 22.03.2 for yoga (UCA for ubuntu 20.04) as well.

Ammad Ali

[James Page (james-page)]

Still pending acceptance of bug 2003060 into proposed.

[Ammad Ali (syedammad83)]

Bug https://bugs.launchpad.net/bugs/2003060 has been fixed. The new OVN 22.03.2 will fix the critical issue related to OVN QoS with neutron.

[Ammad Ali (syedammad83)]

I have tested this with openstack yoga release, and ovn 22.03.2 is working fine.

[Ammad Ali (syedammad83)]

ping

[Frode Nordahl (fnordahl)]

Thank you for your help in testing, we need to provide verification for all the LP bugs mentioned in the changelog on https://launchpad.net/ubuntu/+source/ovn/22.03.2-0ubuntu0.22.04.1 for this to proceed.

I'll endeavor to contribute verification trough exercising the upstream OVN system testsuites with the package today.

[Frode Nordahl (fnordahl)]

ubuntu@vast-yak:~/ovn-22.03.2$ dpkg -l | egrep "(openvswitch|ovn)"
ii openvswitch-common 2.17.5-0ubuntu0.22.04.2 amd64 Open vSwitch common components
ii openvswitch-source 2.17.5-0ubuntu0.22.04.2 all Open vSwitch source code
ii openvswitch-switch 2.17.5-0ubuntu0.22.04.2 amd64 Open vSwitch switch implementations
ii ovn-central 22.03.2-0ubuntu0.22.04.1 amd64 OVN central components
ii ovn-common 22.03.2-0ubuntu0.22.04.1 amd64 OVN common components
ii ovn-host 22.03.2-0ubuntu0.22.04.1 amd64 OVN host components
ii python3-openvswitch 2.17.5-0ubuntu0.22.04.2 all Python 3 bindings for Open vSwitch
#
# Setting the AUTOTEST_PATH will ensure binaries from system path, i.e. the package,
# will be used in the test.
#
# For any test listed with FAILURE below it is also rechecked at the end.
#
# Skipped tests are either because of missing test dependency in Ubuntu, or a test
# that is broken for other reasons and that has not regressed since the previous
# package version.
#
ubuntu@vast-yak:~/ovn-22.03.2$ make check-kernel AUTOTEST_PATH=ovs/tests TESTSUITEFLAGS="-j1" SUDO=sudo RECHECK=yes
make all-am
make[1]: Entering directory '/home/ubuntu/ovn-22.03.2'
make[1]: Leaving directory '/home/ubuntu/ovn-22.03.2'
set /bin/bash './tests/system-kmod-testsuite' -C tests AUTOTEST_PATH='ovs/tests'; \
sudo "$@" -j1 -j1 || (test X'yes' = Xyes && sudo "$@" --recheck)
## ----------------------- ##
## ovn 22.03.2 test suite. ##
## ----------------------- ##

system-ovn

  1: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=yes -- parallelization=yes -- ovn_monitor_all=yes ok
  2: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=yes -- parallelization=yes -- ovn_monitor_all=no ok
  3: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=yes -- parallelization=no -- ovn_monitor_all=yes ok
  4: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=yes -- parallelization=no -- ovn_monitor_all=no ok
  5: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=no -- parallelization=yes -- ovn_monitor_all=yes ok
  6: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=no -- parallelization=yes -- ovn_monitor_all=no ok
  7: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=no -- parallelization=no -- ovn_monitor_all=yes ok
  8: 2 LRs connected via LS, gateway router, SNAT and DNAT -- ovn-northd -- dp-groups=no -- parallelization=no -- ovn_monitor_all=no ok
  9: 2 LRs connected via LS, gateway router, SNAT and DNAT - IPv6 -- ovn-northd -- dp-groups=yes -- parallelization=yes -- ovn_monitor_all=yes ok
 10: 2 LRs connected via LS, gateway router, SNAT and DNAT - IPv6 -- ovn-northd -- dp-groups=yes -- parallelization=yes -- ovn_monitor_all=no ok
 11: 2 LRs conne...

[Frode Nordahl (fnordahl)]

I could use some help verifying bug 1971178, it should be trivial to verify and I'm out of runway for this week.

All other bugs referenced in the changelog of this package have now been verified for Jammy.

[Launchpad Janitor (janitor)]

This bug was fixed in the package ovn - 22.03.2-0ubuntu0.22.04.1

---------------
ovn (22.03.2-0ubuntu0.22.04.1) jammy; urgency=medium

  * New upstream version 22.03.2 (LP: #2003056).
  * d/p/lp-1982454-northd-stateless-nat-fix.patch: Drop, included in point release.
  * d/control: Bump openvswitch build requirement.
  * d/flaky-tests.txt: Extend list of flaky tests.

ovn (22.03.1-0ubuntu2) jammy; urgency=medium

  * d/rules, d/ovn-common.install: Properly fix the ovn-detrace binary
    (LP: #1971178).
  * d/gbp.conf: Add the package components.

ovn (22.03.1-0ubuntu1) jammy; urgency=medium

  [ Frode Nordahl ]
  * New upstream point release (LP: #1980809).
  * d/ovn-common.install: Add missing ovn_detrace.py binary (LP: #1971178).
  * d/rules, d/testlist.py, d/flaky-tests.txt:
    - Dynamically build list of tests to run from list of test descriptions.
  * d/p/lp-1982454-northd-stateless-nat-fix.patch (LP: #1982454).

  [ James Page ]
  * d/control: Update openvswitch build requirement.

 -- Frode Nordahl <email address hidden> Tue, 17 Jan 2023 09:45:31 +0100

[Robie Basak (racb)]

The verification of the Stable Release Update for ovn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for ovn has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package ovn - 22.03.2-0ubuntu0.22.04.1~cloud0
---------------

 ovn (22.03.2-0ubuntu0.22.04.1~cloud0) focal-yoga; urgency=medium
 .
   * New upstream release for the Ubuntu Cloud Archive.
 .
 ovn (22.03.2-0ubuntu0.22.04.1) jammy; urgency=medium
 .
   * New upstream version 22.03.2 (LP: #2003056).
   * d/p/lp-1982454-northd-stateless-nat-fix.patch: Drop, included in point release.
   * d/control: Bump openvswitch build requirement.
   * d/flaky-tests.txt: Extend list of flaky tests.
 .
 ovn (22.03.1-0ubuntu2) jammy; urgency=medium
 .
   * d/rules, d/ovn-common.install: Properly fix the ovn-detrace binary
     (LP: #1971178).
   * d/gbp.conf: Add the package components.
 .
 ovn (22.03.1-0ubuntu1) jammy; urgency=medium
 .
   [ Frode Nordahl ]
   * New upstream point release (LP: #1980809).
   * d/ovn-common.install: Add missing ovn_detrace.py binary (LP: #1971178).
   * d/rules, d/testlist.py, d/flaky-tests.txt:
     - Dynamically build list of tests to run from list of test descriptions.
   * d/p/lp-1982454-northd-stateless-nat-fix.patch (LP: #1982454).
 .
   [ James Page ]
   * d/control: Update openvswitch build requirement.

[Corey Bryant (corey.bryant)]

Hello Frode, or anyone else affected,

Accepted ovn into ovn-22.03-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:ovn-22.03-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-ovn-22.03-needed to verification-ovn-22.03-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-ovn-22.03-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Moved back to triaged for ovn-22.03-proposed for the moment. There is a hash mismatch in the cloud archive because I didn't copy the package binaries over.

[Edward Hope-Morley (hopem)]

I have deployed Openstack Focal Ussuri using the focal-22.03/updates and upgraded all nodes to focal-22.03/proposed. I tested connectivity to existing and new vms and all seems good.

[Alan Baghumian (alanbach)]

I just upgraded my whole entire Focal/Yoga cloud to use the new versions.

A few notes:

1. The add repository command did not work, so I just manually added it:

$ sudo add-apt-repository cloud-archive:ovn-22.03-proposed
'ovn-22.03-proposed': not a valid cloud-archive name

$ echo "deb http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03 main" | sudo tee /etc/apt/sources.list.d/cloud-archive-ovn.list; sudo apt-get update; sudo apt-get -y dist-upgrade

2. All my machines are fully patched and the only packages that upgraded were the fresh OVN as well as OpenVSwitch.

3. Upgraded ovn-central units first and restart OVN services.

4. Upgraded ovn-chassis then octvia-ovn-chassis units.

5. Running floating IP ping from existing instances did not report any interruptions.

6. Stopped and started a couple instance and tested their floating IP(s).

7. Launched two new instances in two separate AZ(s), assigned floating IP(s), all good!

Thank you so much for taking care of this.

Best,
Alan

[Corey Bryant (corey.bryant)]

ovn-22.03 regresion was verified successfully with the focal-ussuri-ovn-22.03 tests from charmed-openstack-tester.

======
Totals
======
Ran: 103 tests in 1010.0583 sec.
 - Passed: 87
 - Skipped: 14
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 2
Sum of execute time for each test: 1131.1492 sec.

The 2 failures are known policy issues with designate and unrelated to this change.

$ juju run --application ovn-central 'apt policy ovn-common' | grep 22.03
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages

$ juju run --application ovn-chassis 'apt policy ovn-common' | grep 22.03
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages
      Installed: 22.03.2-0ubuntu0.22.04.1~cloud1
      Candidate: 22.03.2-0ubuntu0.22.04.1~cloud1
     *** 22.03.2-0ubuntu0.22.04.1~cloud1 500
            500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-proposed/ovn-22.03/main amd64 Packages

[Edward Hope-Morley (hopem)]

this is now released to focal-updates/ovn-22.03 so marking at such.