[SRU] Stable point releases for python-ovsdbapp

[Diko Parvanov]

[Impact]
This release sports mostly bug-fixes and we would like to make
sure all of our supported customers have access to these
improvements. The update contains the following package updates:

python-ovsdbapp 2.1.1 (zed)
python-ovsdbapp 1.15.3 (jammy/yoga)
python-ovsdbapp 1.12.3 (xena)
python-ovsdbapp 1.9.4 (wallaby)

Delta between 2.1.0 and 2.1.1 (zed):

 - [97e738d] Fix TRY_AGAIN handling
 - [bbdc14a] Don't force_reconnect() on unhandled Idl exception
 - [c9e9473] Fix logging exception while handling transaction exception
 - [d90358c] Update TOX_CONSTRAINTS_FILE for stable/zed
 - [b58a340] Update .gitreview for stable/zed

Delta between 1.15.1 and 1.15.3 (jammy/yoga):

 - [ab3e0cb] Don't force_reconnect() on unhandled Idl exception
 - [92cbba4] Fix logging exception while handling transaction exception
 - [315c809] Fix TRY_AGAIN handling
 - [a69430a] Handle OVS 2.17 change to Idl.tables
 - [de473ce] Fix broken behavior after recent OVS update
 - [10539ed] github no longer accepts git:// connections
 - [7735d5c] Update TOX_CONSTRAINTS_FILE for stable/yoga
 - [253afd4] Update .gitreview for stable/yoga

Delta between 1.12.0 and 1.12.3 (xena):

 - [8202d53] Don't force_reconnect() on unhandled Idl exception
 - [1b04058] Fix logging exception while handling transaction exception
 - [fc62ae2] Fix TRY_AGAIN handling
 - [4d4402c] Update QoS register in "QoSAddCommand" if exists
 - [62ee947] Add cooperative_yield() to OvsdbIdl
 - [ab571f4] Handle OVS 2.17 change to Idl.tables
 - [a7a1d5c] Fix broken behavior after recent OVS update
 - [a73fbf1] github no longer accepts git:// connections
 - [3a0cd11] Log what exception caused abort of the transaction
 - [8c09568] Update TOX_CONSTRAINTS_FILE for stable/xena
 - [19a4364] Update .gitreview for stable/xena
 - [49ab130] tools: run functional tests against ovn/main branch

Delta between 1.9.0 and 1.9.4 (wallaby):

 - [65d02f0] Don't force_reconnect() on unhandled Idl exception
 - [87dcab2] Fix logging exception while handling transaction exception
 - [dd7e332] Fix TRY_AGAIN handling
 - [f804411] Update QoS register in "QoSAddCommand" if exists
 - [e1a0d7c] Add cooperative_yield() to OvsdbIdl
 - [c6e2c43] Handle OVS 2.17 change to Idl.tables
 - [9990509] Fix broken behavior after recent OVS update
 - [fb5feee] github no longer accepts git:// connections
 - [fa4a747] Log what exception caused abort of the transaction
 - [eaa6406] tools: run functional tests against ovn/main branch
 - [8304251] Actually close the connection in Connection.stop()
 - [66918f9] Don't spam retries 100s of times a second
 - [d0cc492] Add an active wait in the "Backend.lookup"
 - [007c0f6] Fix docs job
 - [f6a19a3] Update TOX_CONSTRAINTS_FILE for stable/wallaby
 - [875be60] Update .gitreview for stable/wallaby

[Test Case]

The microrelease testing documentation at https://wiki.ubuntu.com/StableReleaseUpdates requests 4 points of testing, shown below, along with my comments on the coverage that we currently have in place:

* a reliable and credible test suite for assuring the quality of every commit or release,

coreycb: Upstream has unit tests and they are run during the package build.

* the tests are covering both functionality and API/ABI stability,

coreycb: This would be covered by unit tests as well as our planned functional testing described at [1].

* the tests run during package build to cover all architectures,

coreycb: Unit tests are run during package builds. Since this is a pure python package it is only built on amd64.

* the package has an autopkgtest to run the tests in an Ubuntu environment against the actual binary packages"

coreycb: The autopkgtests only test python imports. Since the regression testing described in [1] will be exercising the actual binary package functionality, I would like to see if we can get approval based on that being an adequate alternative.

[1]
The following SRU process was followed:
https://wiki.ubuntu.com/OpenStackUpdates

In order to avoid regression of existing consumers, the OpenStack team
will run their continuous integration test against the packages that
are in -proposed. A successful run of all available tests will be
required before the proposed packages can be let into -updates.

The OpenStack team will be in charge of attaching the output summary
of the executed tests. The OpenStack team members will not mark
‘verification-done’ until this has happened.

[Regression Potential]
In order to mitigate the regression potential, the results of the
aforementioned tests are attached to this bug.

[Discussion]
I would like to provide these new stable releases via microrelease SRU updates: https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases

The recommended releases that upstream tests with are listed at:
https://github.com/openstack/requirements/blob/stable/yoga/upper-constraints.txt
https://github.com/openstack/requirements/blob/stable/xena/upper-constraints.txt
https://github.com/openstack/requirements/blob/stable/wallaby/upper-constraints.txt

[Original Bug Description]

Subject: ovsdbapp: AttributeError: 'TableSchema' object has no attribute 'condition_state'

After upgrade from victoria to wallaby neutron-api units firing errors. Exception is related to https://bugs.launchpad.net/ovsdbapp/+bug/1972989, which is fixed in ovsdbapp 1.9.3

https://releases.openstack.org/wallaby/ states that ovsdbapp should be 1.9.4 (latest for walalby)

Installed in the unit is:

root@juju-4c2163-3-lxd-14:~# python3 -m pip list | grep ovsdbapp
ovsdbapp 1.9.0

Also:
root@juju-4c2163-3-lxd-14:~# apt-cache policy python3-ovsdbapp
python3-ovsdbapp:
  Installed: 1.9.0-0ubuntu1~cloud0
  Candidate: 1.9.0-0ubuntu1~cloud0
  Version table:
 *** 1.9.0-0ubuntu1~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1.0-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
     1.1.0-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

So it looks like the charm doesn't upgrade ovsdbapp during openstack upgrade, because a newer version with the above fix doesn't exist in the UCA:

 python3-ovsdbapp | 1.9.0-0ubuntu1~cloud0 | wallaby | focal-updates | all
 python3-ovsdbapp | 1.9.0-0ubuntu1~cloud0 | wallaby-proposed | focal-proposed | all

Full traceback:

2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovsdbapp/backend/ovs_idl/connection.py", line 81, in start
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn idlutils.wait_for_change(self.idl, self.timeout)
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/impl_idl_ovn.py", line 52, in wait_for_change
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn while idl_.change_seqno == seqno and not idl_.run():
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 541, in run
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn self.__send_monitor_request()
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn File "/usr/lib/python3/dist-packages/ovs/db/idl.py", line 850, in __send_monitor_request
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn not ConditionState.is_true(table.condition_state.acked)):
2023-02-21 07:39:19.639 431391 ERROR neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.impl_idl_ovn AttributeError: 'TableSchema' object has no attribute 'condition_state'

Worked around by:
juju run -a neutron-api 'sudo python3 -m pip install ovsdbapp==1.9.4'
juju run -a neutron-api 'sudo service neutron-* restart'

[Diko Parvanov (dparv)]

This is also valid for xena and yoga:

 python3-ovsdbapp | 1.12.0-0ubuntu1~cloud0 | xena | focal-updates | all
 python3-ovsdbapp | 1.12.0-0ubuntu1~cloud0 | xena-proposed | focal-proposed | all
 python3-ovsdbapp | 1.15.1-0ubuntu2~cloud0 | yoga | focal-updates | all
 python3-ovsdbapp | 1.15.1-0ubuntu2~cloud0 | yoga-proposed | focal-proposed | all

fixed is contained in 1.12.2 and 1.16.2, this needs to be in UCA

[James Page (james-page)]

Marking the charm task as invalid - the package does get upgraded but it does not have the required fix for the underlying issue.

[James Page (james-page)]

Note that the impacted deployment is using the OVN 22.03 Cloud Archive which pulls in OVS 2.17.3

[James Page (james-page)]

Note that the 1.15.x series aligns with Yoga but has not had a new stable point release including this fix.

[Felipe Reyes (freyes)]

it is in a release now.

$ git tag --contains a69430a967bfa1096b9258a8445b617b9e122a22
1.15.3

[Eric Chen (eric-chen)]

Is there any update?

[Alex Kavanagh (ajkavanagh)]

@Eric

> Is there any update?

Could you be a little more specific, please, in your question?

[Billy Olsen (billy-olsen)]

I think the question is about whether or not an SRU has been targeted to pick this up. The short answer is not yet.

[Eric Chen (eric-chen)]

Do we have plan to apply it into SRU? thanks!

[Corey Bryant (corey.bryant)]

New point releases for python-ovsdbapp have been uploaded to wallaby-staging, xena-staging, jammy unapproved, and zed-staging. Once this is accepted into jammy-proposed I will promote the other packages from <uca>-staging to <uca>-proposed.

[Steve Langasek (vorlon)]

> The following SRU process was followed:
> https://wiki.ubuntu.com/OpenStackUpdates

python-ovsdbapp is not a source package listed as part of that exception.

[Corey Bryant (corey.bryant)]

On Fri, Jul 7, 2023 at 4:25 PM Steve Langasek <email address hidden>
wrote:

> > The following SRU process was followed:
> > https://wiki.ubuntu.com/OpenStackUpdates
>
> python-ovsdbapp is not a source package listed as part of that
> exception.
>
>
Thanks for taking a look. I was hoping to provide the stable releases via
the microrelease SRU exception. Let me know what you think.

Corey

> --
> You received this bug notification because you are subscribed to the bug
> report.
> Matching subscriptions: openstack-charms
> https://bugs.launchpad.net/bugs/2007919
>
> Title:
> [SRU] Stable point releases for python-ovsdbapp
>
> Status in OpenStack Neutron API Charm:
> Invalid
> Status in Ubuntu Cloud Archive:
> Invalid
> Status in Ubuntu Cloud Archive wallaby series:
> Triaged
> Status in Ubuntu Cloud Archive xena series:
> Triaged
> Status in Ubuntu Cloud Archive yoga series:
> Triaged
> Status in Ubuntu Cloud Archive zed series:
> Triaged
> Status in python-ovsdbapp package in Ubuntu:
> Invalid
> Status in python-ovsdbapp source package in Jammy:
> Triaged
>
> Bug description:
> [Impact]
> This release sports mostly bug-fixes and we would like to make
> sure all of our supported customers have access to these
> improvements. The update contains the following package updates:
>
> python-ovsdbapp 2.1.1 (zed)
> python-ovsdbapp 1.15.3 (jammy/yoga)
> python-ovsdbapp 1.12.3 (xena)
> python-ovsdbapp 1.9.4 (wallaby)
>
> Delta between 2.1.0 and 2.1.1 (zed):
>
> - [97e738d] Fix TRY_AGAIN handling
> - [bbdc14a] Don't force_reconnect() on unhandled Idl exception
> - [c9e9473] Fix logging exception while handling transaction exception
> - [d90358c] Update TOX_CONSTRAINTS_FILE for stable/zed
> - [b58a340] Update .gitreview for stable/zed
>
> Delta between 1.15.1 and 1.15.3 (jammy/yoga):
>
> - [ab3e0cb] Don't force_reconnect() on unhandled Idl exception
> - [92cbba4] Fix logging exception while handling transaction exception
> - [315c809] Fix TRY_AGAIN handling
> - [a69430a] Handle OVS 2.17 change to Idl.tables
> - [de473ce] Fix broken behavior after recent OVS update
> - [10539ed] github no longer accepts git:// connections
> - [7735d5c] Update TOX_CONSTRAINTS_FILE for stable/yoga
> - [253afd4] Update .gitreview for stable/yoga
>
> Delta between 1.12.0 and 1.12.3 (xena):
>
> - [8202d53] Don't force_reconnect() on unhandled Idl exception
> - [1b04058] Fix logging exception while handling transaction exception
> - [fc62ae2] Fix TRY_AGAIN handling
> - [4d4402c] Update QoS register in "QoSAddCommand" if exists
> - [62ee947] Add cooperative_yield() to OvsdbIdl
> - [ab571f4] Handle OVS 2.17 change to Idl.tables
> - [a7a1d5c] Fix broken behavior after recent OVS update
> - [a73fbf1] github no longer accepts git:// connections
> - [3a0cd11] Log what exception caused abort of the transaction
> - [8c09568] Update TOX_CONSTRAINTS_FILE for stable/xena
> - [19a4364] Update .gitreview for stable/xena
> - [49ab130] tools: run functional tests against ovn/main branch
>
> Delta between 1.9.0 and 1.9.4 (wallaby):
>
> - [65d02f0] Don't force_reconnect()...

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted python-ovsdbapp into zed-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:zed-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-zed-needed to verification-zed-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-zed-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted python-ovsdbapp into xena-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:xena-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-xena-needed to verification-xena-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-xena-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Corey Bryant (corey.bryant)]

Hello Diko, or anyone else affected,

Accepted python-ovsdbapp into wallaby-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:wallaby-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-wallaby-needed to verification-wallaby-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-wallaby-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

> I would like to provide these new stable releases via microrelease SRU updates: https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases

Please provide the information specified at that link.

The autopkgtest doesn't look like it's running the test suite. Is it?

[Robie Basak (racb)]

> The autopkgtest doesn't look like it's running the test suite. Is it?

It's also reporting neutral, rather than pass, probably as a consequence of the only test being marked "superficial". According to https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases, a dep8 test is a requirement. Please could you look in to this?

[Corey Bryant (corey.bryant)]

Thanks Robie. I've updated the [Test Case] section with more testing details.

[Robie Basak (racb)]

Thanks.

I looked at the specific changes being made by upstream in this microrelease, and have some concerns. I found them broken down by commit here: https://opendev.org/openstack/ovsdbapp/commits/branch/stable/yoga

https://opendev.org/openstack/ovsdbapp/commit/ab3e0cb0d0865417efbf103f44954573a5ba92ac - this seems like a deliberate change in behaviour. But it doesn't come with any tests, and there's no explanation as to why it's needed in a stable release. Is this even a bugfix? What if the retrying behaviour *is* working for some user? Won't dropping that behaviour regress them?

https://opendev.org/openstack/ovsdbapp/commit/92cbba4481a7c7acedf24d8250fc7cf4be6b46d7 seems OK as a bugfix, but doesn't come with a test.

https://opendev.org/openstack/ovsdbapp/commit/315c8096c9bb3ad6d09e3b2f09bc1c128bc35497 seems good as a bugfix, and refers to bug 1988457 as what it is fixing, but doesn't come with a test.

https://opendev.org/openstack/ovsdbapp/commit/a69430a967bfa1096b9258a8445b617b9e122a22 doesn't come with a test.

https://opendev.org/openstack/ovsdbapp/commit/de473cea9defe04a6f904fa63a8cf49746534916 fixes a regression caused by a change in OVS, and does come with a test - great!

Conclusion

It doesn't seem like upstream microrelease updates come with enough testing to land them for Ubuntu users in stable releases without further manual testing. The upstream ab3e0cb commit is also concerning as it is a deliberate change in behaviour.

NACK for an Ubuntu SRU based on the upstream microrelease exception. Please justify and test each change you need individually instead.

[Robie Basak (racb)]

An upload of python-ovsdbapp to jammy-proposed has been rejected from the upload queue for the following reason: "See https://bugs.launchpad.net/charm-neutron-api/+bug/2007919/comments/19".

[Corey Bryant (corey.bryant)]

Thanks for the review Robie. I appreciate your thorough reviews and keeping Ubuntu stable!

(Note: The following reply is taken from my reply to a related thread: https://lists.ubuntu.com/archives/ubuntu-release/2023-September/005765.html)

For the behavioral change [1], I think there is minimal regression potential as the change is limited to an exception path where sleeps and reconnects were determined to be unnecessary. I'd have appreciated it if they would have provided a bug reference for the commit to provide more context, however with that said, it seems they are making a code improvement here for performance reasons and they found it to be useful enough to backport to stable branches.
[1] https://opendev.org/openstack/ovsdbapp/commit/ab3e0cb0d0865417efbf103f44954573a5ba92ac

It is certainly not ideal to have missing unit tests. It's important to consider that we perform testing above and beyond the unit tests that are run as part of our package builds. The regression testing that we have in place for OpenStack is run for all stable releases. For that we deploy a full OpenStack cloud and then run tempest integration tests that exercise all of the packages and the cloud solution as a whole.

I think we have a solid history of limiting regression potential for OpenStack users in our stable releases and that remains very important to us.

[Gustavo Sanchez (gustavosr98)]

Subscribing to "field-critical"

It is 1. affecting an FE Openstack upgrades project on Openstack Wallaby, Xena and Yoga releases
It is 2. blocking a current FE upgrades project to upgrade to Yoga since there is no package with the patch available on any pocket for Yoga

[Zhanglei Mao (zhanglei-mao)]

Feedback for #15
It didn't work for me when upgrade from v->w with
 juju config neutron-api cloud:focal-wallaby/proposed

PIP install still needs
  sudo python3 -m pip install ovsdbapp==1.9.4'
  sudo systemctl restart “neutron*”

[Felipe Reyes (freyes)]

On Sun, 2024-01-07 at 14:23 +0000, Zhanglei Mao wrote:
> Feedback for #15
> It didn't work for me when upgrade from v->w with
>  juju config neutron-api cloud:focal-wallaby/proposed
>
> PIP install still needs
>   sudo python3 -m pip install ovsdbapp==1.9.4'
>   sudo systemctl restart “neutron*”
>

The 1.9.4 version is in wallaby-proposed.

 python3-ovsdbapp | 1.9.4-0ubuntu1~cloud0 | wallaby-proposed | focal-proposed | all

Can you share the output of "apt policy python3-ovsdbapp"?