Version 0.6.7-3 is broken for "del" operations on recent kernels

[Eugene Crosser]

[Impact]

Because of this bug upstream https://github.com/svinota/pyroute2/issues/923, in the versions of pyroute2 prior to 0.6.10, all delete operations (address, link, route, ...) fail with 'Operation not supported'.

This is related to the kernel commit https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=a6cec0bcd34264be8887791594be793b3f12719f

Pyroute package has to be upgraded to a version 0.6.10 or later as long as the distro comes with the kernel that contains the above commit.

Reproducer:

$ sudo python3
Python 3.10.7 (main, Sep 8 2022, 14:34:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyroute2 import IPRoute
>>> IPRoute().link("add", ifname="dum0", kind="dummy")
({'header': {'length': 36, 'type': 2, 'flags': 256, 'sequence_number': 255, 'pid': 4993, 'error': None, 'target': 'localhost', 'stats': Stats(qsize=0, delta=0, delay=0)}, 'event': 'NLMSG_ERROR'},)
>>> IPRoute().link_lookup(ifname="dum0")
[10]
>>> IPRoute().link("del", ifname="dum0")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pr2modules/iproute/linux.py", line 1489, in link
    ret = self.nlm_request(msg, msg_type=command, msg_flags=msg_flags)
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 401, in nlm_request
    return tuple(self._genlm_request(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 906, in nlm_request
    for msg in self.get(
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 404, in get
    return tuple(self._genlm_get(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 732, in get
    raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (95, 'Operation not supported')

ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: python3-pyroute2 0.6.7-3
ProcVersionSignature: Ubuntu 5.19.0-23.24-generic 5.19.7
Uname: Linux 5.19.0-23-generic x86_64
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
CasperMD5CheckResult: unknown
CurrentDesktop: ubuntu:GNOME
Date: Wed Nov 2 10:44:29 2022
InstallationDate: Installed on 2021-08-18 (440 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
PackageArchitecture: all
SourcePackage: pyroute2
UpgradeStatus: Upgraded to kinetic on 2022-10-23 (9 days ago)

[Test Case]

There is a small reproducer above that Eugene included in the bug report.

Additionally we can use the standard charm-deployed regression test suite to test kinetic-zed support. nova-compute units can be inspected to see if the 'Operation not supported' errors are no longer produced in /var/log/neutron/neutron-ovn-metadata-agent.log.

[Regression Potential]

The patch has already landed in releases of pyroute2 0.6.10 and above. Since pyroute2 is also backported to the zed cloud archive (which is on jammy) we'll need to ensure that this patch behaves as expected with jammy-zed deploys as well.

[Corey Bryant (corey.bryant)]

Eugene, Thank you for reporting this bug.

[Corey Bryant (corey.bryant)]

A new package version with this fix has been uploaded to the kinetic unapproved queue where it will await SRU team review.
https://launchpad.net/ubuntu/kinetic/+queue?queue_state=1&queue_text=pyroute2

[Brian Murray (brian-murray)]

Hello Eugene, or anyone else affected,

Accepted pyroute2 into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pyroute2/0.6.7-3ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Eugene Crosser (crosser)]

0.6.7-3ubuntu1 from kinetic-proposed fixes the problem for me in the real use case where it was discovered (as well as in the reproducer included in the original report).

(Ramble: I wish package maintenance kept up to date with the upstream. Version chosen by openstack maintainers is rather old, and has many features missing, as well as potentially some bugfixes...)

[Launchpad Janitor (janitor)]

This bug was fixed in the package pyroute2 - 0.6.7-3ubuntu1

---------------
pyroute2 (0.6.7-3ubuntu1) kinetic; urgency=medium

  * d/p/iproute-linux-try-to-improve-flags-when-sending-del-msgs.patch:
    Cherry-picked from upstream 0.6.10 to fix delete operations that
    fail with 'Operation not supported' (LP: #1995469).

 -- Corey Bryant <email address hidden> Wed, 23 Nov 2022 12:47:27 -0500

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for pyroute2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

Regression testing successful for jammy-zed (3 failures are known/unrelated issues):

jammy-zed
---------
======
Totals
======
Ran: 186 tests in 487.0430 sec.
 - Passed: 171
 - Skipped: 12
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 3
Sum of execute time for each test: 557.0849 sec.

[Corey Bryant (corey.bryant)]

The verification of the Stable Release Update for pyroute2 has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Corey Bryant (corey.bryant)]

This bug was fixed in the package pyroute2 - 0.6.7-3ubuntu1~cloud0
---------------

 pyroute2 (0.6.7-3ubuntu1~cloud0) jammy-zed; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 pyroute2 (0.6.7-3ubuntu1) kinetic; urgency=medium
 .
   * d/p/iproute-linux-try-to-improve-flags-when-sending-del-msgs.patch:
     Cherry-picked from upstream 0.6.10 to fix delete operations that
     fail with 'Operation not supported' (LP: #1995469).

[Trent Lloyd (lathiat)]

This also effects jammy-yoga with the current hwe-edge 5.19.0-28.

Seems like we should actually backport this to jammy itself.

[DUFOUR Olivier (odufourc)]

subscribed ~field-critical

Like Trent mentioned previously, this has an impact as well on Jammy Yoga using the HWE kernel.

This has been seen and reproduced on 2 different environments and is blocking the deployment on Prodstack6 where, due to driver constraint, using the HWE kernel is an absolute requirement.

[DUFOUR Olivier (odufourc)]

Workaround to manually install the python3-pyroute2 packages from the Archive tracker on Jammy-Zed seems to fix temporarily the issue as well

To download the packages from one machine with Zed repository enabled on Jammy :

for package in $(apt search python3-pyroute2 2>/dev/null | grep installed | cut -f 1 -d '/'); do apt download $package; done
$ ls pyroute2/ -l
total 432
-rw-r--r-- 1 ubuntu ubuntu 171920 Feb 20 05:33 python3-pyroute2.core_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 23300 Feb 20 05:33 python3-pyroute2.ethtool_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 52456 Feb 20 05:33 python3-pyroute2.ipdb_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 25964 Feb 20 05:33 python3-pyroute2.ipset_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 66530 Feb 20 05:33 python3-pyroute2.ndb_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 20944 Feb 20 05:33 python3-pyroute2.nftables_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 24546 Feb 20 05:33 python3-pyroute2.nslink_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 21990 Feb 20 05:33 python3-pyroute2.protocols_0.6.7-3ubuntu1~cloud0_all.deb
-rw-r--r-- 1 ubuntu ubuntu 18656 Feb 20 05:33 python3-pyroute2_0.6.7-3ubuntu1~cloud0_all.deb

Install all the following packages on all nova-computes :
for i in {0..2}; do juju scp -- -r ./pyroute2 $i:~/ ; juju ssh $i sudo dpkg -i ./pyroute2/* ; done
juju run -a nova-compute systemctl restart neutron-ovn-metadata-agent.service

[James Page (james-page)]

Raised bug task for jammy and uploaded for SRU team to review.

Note that this only impacts deployments where the HWE kernel is in use; I've tested the proposed package both with the release kernel and the HWE kernel on 22.04.

[Robie Basak (racb)]

Hello Eugene, or anyone else affected,

Accepted pyroute2 into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/pyroute2/0.6.4-3ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[James Page (james-page)]

Reconfirmed the original bug:

root@bug1995469:~# python3 test.py
Traceback (most recent call last):
  File "/root/test.py", line 4, in <module>
    IPRoute().link("del", ifname="dum0")
  File "/usr/lib/python3/dist-packages/pr2modules/iproute/linux.py", line 1393, in link
    ret = self.nlm_request(msg,
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 393, in nlm_request
    return tuple(self._genlm_request(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 885, in nlm_request
    for msg in self.get(msg_seq=msg_seq,
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 396, in get
    return tuple(self._genlm_get(*argv, **kwarg))
  File "/usr/lib/python3/dist-packages/pr2modules/netlink/nlsocket.py", line 721, in get
    raise msg['header']['error']
pr2modules.netlink.exceptions.NetlinkError: (95, 'Operation not supported')

and verified with the package in proposed:

root@bug1995469:~# python3 test.py
<command successfully completes>
root@bug1995469:~# apt-cache policy python3-pyroute2
python3-pyroute2:
  Installed: 0.6.4-3ubuntu2
  Candidate: 0.6.4-3ubuntu2
  Version table:
 *** 0.6.4-3ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.6.4-3ubuntu1.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages
     0.6.4-3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

[Andreas Hasenack (ahasenack)]

I verified the test results and am satisfied that they show the executed planned test case, and that the results are correct.

The package built correctly in all architectures and Ubuntu releases it was meant for.

There are no DEP8 regressions, or they were fixed.

There is no SRU freeze ongoing at the moment.

There is no halted phasing on a previous update.

[Launchpad Janitor (janitor)]

This bug was fixed in the package pyroute2 - 0.6.4-3ubuntu2

---------------
pyroute2 (0.6.4-3ubuntu2) jammy; urgency=medium

  * d/p/iproute-linux-try-to-improve-flags-when-sending-del-msgs.patch:
    Cherry-picked from upstream 0.6.10 to fix delete operations that
    fail with 'Operation not supported' (LP: #1995469).

 -- James Page <email address hidden> Mon, 20 Feb 2023 09:55:21 +0000

[Boris Lukashev (rageltman)]

Focal is in-need of this package as well - OpenStack Yoga can run on Focal and this is breaking the OVN metadata service (at least).

[Maksim Malchuk (mmalchuk)]

This package already exist in focal-yoga of the Ubuntu Cloud Archive so Focal users can use fix.

----
pyroute2 (0.6.4-3ubuntu2~cloud0) focal-yoga; urgency=medium

  * New update for the Ubuntu Cloud Archive.

 -- Openstack Ubuntu Testing Bot <email address hidden> Thu, 30 Mar 2023 15:53:35 +0000

pyroute2 (0.6.4-3ubuntu2) jammy; urgency=medium

  * d/p/iproute-linux-try-to-improve-flags-when-sending-del-msgs.patch:
    Cherry-picked from upstream 0.6.10 to fix delete operations that
    fail with 'Operation not supported' (LP: #1995469).

 -- James Page <email address hidden> Mon, 20 Feb 2023 09:55:21 +0000