pyroute2 0.5.2 doesn't support neutron-common 14.0.4

Bug #1861457 reported by Ryan Farrell on 2020-01-30
34
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack neutron-gateway charm
Undecided
Unassigned
Ubuntu Cloud Archive
Undecided
Unassigned
Stein
High
James Page
neutron (Ubuntu)
Undecided
James Page

Bug Description

On a Bionic-Stein clouds we recently updated all packages and encountered issues creating routers namespaces due to some changes in the data structure expected from pyroute2.

For Bionic/Stein, the latest neutron-common package is from 14.0.4 and the latest pyroute2 is 0.5.2.

We were able to resolve the issue by forcing an update of pyroute to 0.5.6 from the Eoan repository on n-gw units and cycling the neutron-openvswitch-agent and neutron-l3-agent services.
Fixes will need to be back ported and released to Bionic Stein repo.

2020-01-30 20:00:49.261 1874349 DEBUG oslo_concurrency.lockutils [-] Releasing lock "iptables-qrouter-5241d017-9d8a-44b4-912f-d36703bbc259" lock /usr/lib/python3/dist-packages/oslo_concurrency/lockutils.py:281
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info [-] 255: KeyError: 255
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 158, in call
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1188, in process
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info self.process_external()
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 946, in process_external
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info self._process_external_gateway(ex_gw_port)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info self.external_gateway_added(ex_gw_port, interface_name)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 434, in external_gateway_added
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info self._plug_external_gateway(ex_gw_port, interface_name, self.ns_name)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info mtu=ex_gw_port.get('mtu'))
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 267, in plug
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info bridge, namespace, prefix, mtu)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 355, in plug_new
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info namespace2=namespace)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 213, in add_veth
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info name1, self.namespace, 'veth', peer=peer)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info return input_func(*args, **kwargs)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 241, in _wrap
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info return self.channel.remote_call(name, args, kwargs)
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 203, in remote_call
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info raise exc_type(*result[2])
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info KeyError: 255
2020-01-30 20:00:49.262 1874349 ERROR neutron.agent.l3.router_info
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: 5241d017-9d8a-44b4-912f-d36703bbc259: KeyError: 255
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 723, in _process_routers_if_compatible
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 567, in _process_router_if_compatible
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self._process_added_router(router)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/agent.py", line 575, in _process_added_router
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent ri.process()
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 485, in process
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent super(HaRouter, self).process()
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 161, in call
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self.logger(e)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self.force_reraise()
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent raise value
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/common/utils.py", line 158, in call
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 1188, in process
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self.process_external()
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 946, in process_external
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self._process_external_gateway(ex_gw_port)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 828, in _process_external_gateway
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self.external_gateway_added(ex_gw_port, interface_name)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/ha_router.py", line 434, in external_gateway_added
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent self._plug_external_gateway(ex_gw_port, interface_name, self.ns_name)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/l3/router_info.py", line 670, in _plug_external_gateway
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent mtu=ex_gw_port.get('mtu'))
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 267, in plug
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent bridge, namespace, prefix, mtu)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/interface.py", line 355, in plug_new
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent namespace2=namespace)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/agent/linux/ip_lib.py", line 213, in add_veth
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent name1, self.namespace, 'veth', peer=peer)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/neutron/privileged/agent/linux/ip_lib.py", line 50, in sync_inner
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent return input_func(*args, **kwargs)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/oslo_privsep/priv_context.py", line 241, in _wrap
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs)
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent File "/usr/lib/python3/dist-packages/oslo_privsep/daemon.py", line 203, in remote_call
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2020-01-30 20:00:49.263 1874349 ERROR neutron.agent.l3.agent KeyError: 255

Felipe Reyes (freyes) wrote :

I *think* this is the patch that we could be missing in 0.5.2 https://github.com/svinota/pyroute2/issues/531 , this is just a hunch, I don't have data to confirm this.

tags: added: sts
Chris Sanders (chris.sanders) wrote :

I've added field-critical due to the risk to production clouds and the only work around currently being to compile/install from outside the cloud archive.

James Page (james-page) wrote :

Neutron expresses:

 pyroute2>=0.5.3;sys_platform!='win32' # Apache-2.0 (+ dual licensed GPL2)

whereas we're baselines on 0.5.2

Changed in charm-neutron-gateway:
status: New → Invalid
James Page (james-page) wrote :

Working on reproducing this issue so we can identify exactly what the issue is.

FTR I'm not adverse to bumping to pyroute2 0.5.3 in the stein UCA if that resolves the issue (as that the minimum version identified in neutron anyway).

Changed in neutron (Ubuntu):
assignee: nobody → James Page (james-page)
James Page (james-page) wrote :

Making the assumption this is probably a ha router (detailed in trace) but details on the exact setup of the deployment would be helpful

James Page (james-page) wrote :

I'm struggling to reproduce this issue - I've tried with HA and non HA routers with bionic/stein but the namespaces are created correctly.

James Page (james-page) on 2020-01-31
Changed in neutron (Ubuntu):
status: New → Incomplete
Ryan Farrell (whereisrysmind) wrote :

Maybe it would help if we describe the flow of upgrades:

The cloud started with neutron packages at 14.0.2 on the computes, n-gw and n-api;
- A charm upgrade for trilio on the computes ran apt-upgrade, which moved the computes to neutron 14.0.4. This exposed a different know issue, which needed to be resolved by moving n-gw and n-api neutron packages to the same rev.

- We performed apt install neutron-common to n-api, and ngw, moving neutron to 14.0.4 there as well.

- Still experiencing networking issues we ran apt-upgrade on n-gw and n-api, and restarted all non cleanup neutron services. Then we observed the log messages posted in this ticket, and finally resolved this issue by forcing pyroute2 to 0.5.6, and removing and re-adding all preexisting ha-routers from their l3-agent.

Erlon R. Cruz (sombrafam) wrote :

Hi Ryan,

What was the issue exposed after trilio's apt-upgrade?
So trilio charm was run on the n-api nodes right? Where all neutron* became 14.04.

So, you upgraded n-gw to 14.04. After that you run apt-upgrade on n-gw and n-api again?
It shouldn't do anything right?

Erlon

David Ames (thedac) wrote :

FYI, Canonical QA has seen this in a duplicate bug with a clean deploy:
https://bugs.launchpad.net/charm-neutron-gateway/+bug/1862200.

It presented as FIPs missing associated iptables NAT rules on the gateway node.

James Page (james-page) on 2020-02-10
Changed in cloud-archive:
status: New → Invalid
Changed in neutron (Ubuntu):
status: Incomplete → Invalid
James Page (james-page) wrote :

OK on the assumption that the minor version mismatch in pyroute2 is the cause of this bug, I've uploaded 0.5.4 to stein-staging and will promote to -proposed.

This is the upper version constraint for OpenStack Stein so keeps up aligned to upstream testing versions.

Hello Ryan, or anyone else affected,

Accepted pyroute2 into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
Jason Hobbs (jason-hobbs) wrote :

I'm getting errors using proposed:
http://paste.ubuntu.com/p/6wMxVZbYqz/

2020-02-10 15:24:38 DEBUG install Setting up python3-pyroute2 (0.5.4-0ubuntu0.19.04.1~cloud0) ...
2020-02-10 15:24:38 DEBUG install update-alternatives: using /usr/bin/python3-ss2 to provide /usr/bin/ss2 (ss2) in auto mode
2020-02-10 15:24:38 DEBUG install update-alternatives: error: alternative path /usr/bin/python3-pyroute2-cli doesn't exist
2020-02-10 15:24:38 DEBUG install dpkg: error processing package python3-pyroute2 (--configure):
2020-02-10 15:24:38 DEBUG install installed python3-pyroute2 package post-installation script subprocess returned error exit status 2

Jason Hobbs (jason-hobbs) wrote :
tags: added: verification-stein-failed
removed: verification-stein-needed
James Page (james-page) wrote :

Hello Ryan, or anyone else affected,

Accepted pyroute2 into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-stein-needed
removed: verification-stein-failed
Jason Hobbs (jason-hobbs) wrote :

I tested with the package from stein-proposed (python3-pyroute2 0.5.4-0ubuntu0.19.04.1~cloud1) and got successful results. The issue I was seeing in bug 1862200 is no longer occurring. I'm marking this verification-stein-done.

tags: added: verification-stein-done
removed: verification-stein-needed
James Page (james-page) wrote :

Also confirmed that the pyroute2 update does not introduce and regressions in the tempest functional test suite we use for SRU testing:

======
Totals
======
Ran: 92 tests in 581.9798 sec.
 - Passed: 84
 - Skipped: 8
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 720.0968 sec.

James Page (james-page) wrote :

ii python3-pyroute2 0.5.4-0ubuntu0.19.04.1~cloud1 all Python3 Netlink library

The verification of the Stable Release Update for pyroute2 has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

James Page (james-page) wrote :

This bug was fixed in the package pyroute2 - 0.5.4-0ubuntu0.19.04.1~cloud1
---------------

 pyroute2 (0.5.4-0ubuntu0.19.04.1~cloud1) bionic; urgency=medium
 .
   * New upstream point release to resolve compatibility issues with
     OpenStack Neutron (LP: #1861457).
   * d/rules,python{3}-pyroute2.{postinst,prerm,postrm}: Use alternatives to
     manage ss2 binary.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.