neutron

l3 agent fails to process router breaking floating ips when updating them on the router qg port via pyroute2

Bug #1991501 reported by sean mooney
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Undecided
Unassigned

Bug Description

2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 639, in _process_router_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 719, in _process_updated_router
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent ri.process()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.logger(e)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.force_reraise()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise self.value
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 174, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.process_external()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return caller(func, *(extras + args), **kw)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return f(*a, **k)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_external_gateway(ex_gw_port)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 931, in _process_external_gateway
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.external_gateway_updated(ex_gw_port, interface_name)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 884, in external_gateway_updated
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._external_gateway_added(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 822, in _external_gateway_added
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._external_gateway_settings(ex_gw_port, interface_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 835, in _external_gateway_settings
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.driver.init_router_port(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 171, in init_router_port
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.init_l3(device_name=device_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 146, in init_l3
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent device.delete_addr_and_conntrack_state(ip_cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 361, in delete_addr_and_conntrack_state
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.addr.delete(cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 540, in delete
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent delete_ip_address(cidr, self.name, self._parent.namespace)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent privileged.delete_ip_address(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3

version info
yoga deoployed with kolla-ansible using debian ubuntu or centos image and popos 22.04 host

popos 22.04 is based on ubuntu 22.04 and uses nftables intead of legacy ip tables

in the contianer i noticed that iptable -L is slow but iptable -S is fast
it is not clare if this is a race or some other issue but intially the router is fine and hte floating ips work and restarting the l3 agent fixes the issue for a time then it breaks with this message.

Tags: ovs l3
sean mooney (sean-k-mooney)
tags: added: l3 ovs
Revision history for this message
sean mooney (sean-k-mooney) wrote (last edit ):
Download full text (3.7 KiB)

just adding this for context if i manually exec into the container as root(so that i can spawn the privsep damon process) and invoke the failing command via python it fails in the same way

sean@cloud:~$ docker exec -it -u root neutron_l3_agent bash
(neutron-l3-agent)[root@cloud /]# python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neutron.agent.linux import ip_lib
>>> ip_lib.delete_ip_address("172.20.0.79/32", "qg-4bb9d20b-a0", namespace="qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
    privileged.delete_ip_address(
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
    raise exc_type(*result[2])
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.

however i can add and remove the IP via the commandline
sudo ip netns exec qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf bash

root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.79/32 brd 172.20.0.79 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a del 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a add 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/e...

Read more...

Revision history for this message
sean mooney (sean-k-mooney) wrote :

i think this might be

https://bugzilla.redhat.com/show_bug.cgi?id=2094986
which is fixed in https://github.com/svinota/pyroute2/commit/1eb08312de30a083bcfddfaa9c1d5e124b6368df

which is released in pyroute2 0.6.10
I'm using 0.6.6 which is the current cap in yoga
https://github.com/openstack/requirements/blob/stable/yoga/upper-constraints.txt#L165

sean mooney (sean-k-mooney)
summary: - l3 agent fails to init router breaking floating ips
+ l3 agent fails to process router breaking floating ips when updating
+ them on the router qg port via iproute
summary: l3 agent fails to process router breaking floating ips when updating
- them on the router qg port via iproute
+ them on the router qg port via pyroute2
Revision history for this message
Brian Haley (brian-haley) wrote :

Right, this looks like a pyroute2 issue which has since been fixed, and neutron bumped it's lower contraints file to use the later version.

Changed in neutron:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.