neutron

neutron-l3-agent traceback while updating router

Bug #1615390 reported by Serguei Bezverkhi
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Expired
Undecided
Unassigned

Bug Description

With the latest master I see this traceback over and over in the log.

2016-08-21 11:08:24.256 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
2016-08-21 11:08:24.258 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 0745147d6f0d4b3380c0da2d53c9909a exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448
2016-08-21 11:08:24.564 1 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 0745147d6f0d4b3380c0da2d53c9909a __call__ /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:296
2016-08-21 11:08:24.565 1 DEBUG neutron.agent.linux.utils [-] Running command: ['ip', '-o', 'link', 'show', 'br-ex'] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-21 11:08:24.724 1 DEBUG neutron.agent.linux.utils [-] Exit code: 0 execute /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:140
2016-08-21 11:08:24.725 1 DEBUG neutron.callbacks.manager [-] Notify callbacks for router, before_update _notify_loop /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/callbacks/manager.py:140
2016-08-21 11:08:24.726 1 DEBUG neutron.agent.l3.router_info [-] process router updates process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py:1036
2016-08-21 11:08:24.727 1 DEBUG neutron.agent.linux.utils [-] Running command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-baa3335b-0013-42dd-856a-64a5c2557a01', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] create_process /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py:83
2016-08-21 11:08:24.988 1 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument

2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info existing_devices = self._get_existing_devices()
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info raise RuntimeError(msg)
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info
2016-08-21 11:08:24.989 1 ERROR neutron.agent.l3.router_info
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'baa3335b-0013-42dd-856a-64a5c2557a01'
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 495, in _process_router_update
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 432, in _process_router_if_compatible
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 447, in _process_updated_router
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent ri.process(self)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 245, in call
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent self.logger(e)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent self.force_reraise()
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/common/utils.py", line 242, in call
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 1037, in process
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 543, in _process_internal_ports
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent existing_devices = self._get_existing_devices()
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 426, in _get_existing_devices
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent ip_devs = ip_wrapper.get_devices(exclude_loopback=True)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 131, in get_devices
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 138, in execute
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent setting the network namespace "qrouter-baa3335b-0013-42dd-856a-64a5c2557a01" failed: Invalid argument
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent
2016-08-21 11:08:24.990 1 ERROR neutron.agent.l3.agent
2016-08-21 11:08:24.992 1 DEBUG neutron.agent.l3.agent [-] Starting router update for baa3335b-0013-42dd-856a-64a5c2557a01, action None, priority 1 _process_router_update /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py:461
2016-08-21 11:08:24.993 1 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: ded128bbbae5422ea5ddc3a2fda4ae1d exchange 'neutron' topic 'q-l3-plugin' _send /var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:448

Revision history for this message
Miguel Lavalle (minsel) wrote :

Hi, Thanks for filing this bug. I re-built my dev system last night and I don't see this traceback in the L3 agent log. Could you provide more information as to under what conditions you get the traceback? What operation are you performing?

Changed in neutron:
status: New → Incomplete
Revision history for this message
Serguei Bezverkhi (sbezverk) wrote :

This happens in Kubernetes environment when neutron-l3-agent runs wihtin a docker container. I suspect the issue is more about ip netns returning RTNETLINK answers: Invalid argument, than the actual l3 agent issue.
Anything you can suggest for further troubleshooting?

ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qdhcp-f32fdde6-bb99-4981-926b-a7df30f0a612
RTNETLINK answers: Invalid argument
qrouter-baa3335b-0013-42dd-856a-64a5c2557a01

This is taking on the host where l3 agent and dhcp containers are running..

Revision history for this message
Matthew Taylor (matthew-taylor-f) wrote :

Just out of curiosity, what are your file perms in /var/run/netns/ when the error pops up?

Revision history for this message
Serguei Bezverkhi (sbezverk) wrote :

I tried to change permissions but it does not help.

[root@k8s-4 ~]# ls -al /var/run/netns
total 0
drwxr-xr-x. 2 root root 160 Sep 7 18:01 .
drwxr-xr-x. 29 root root 880 Sep 7 17:58 ..
----------. 1 root root 0 Sep 7 18:00 qdhcp-04baff87-98c2-4ac0-b8e1-16b0ed692671
----------. 1 root root 0 Sep 7 18:00 qdhcp-2b37f37f-2c19-4759-81ca-1deac5860bfa
----------. 1 root root 0 Sep 7 18:00 qdhcp-5260ce27-36a4-4e93-9771-f9f90b94e057
----------. 1 root root 0 Sep 7 18:00 qdhcp-53ac44ee-5527-4a5e-a521-45ed2ed0231b
----------. 1 root root 0 Sep 7 18:00 qdhcp-977a4d48-1613-457f-b99c-a31870f42858
----------. 1 root root 0 Sep 7 18:01 qrouter-cbf00c95-b3ed-4f21-98fb-c7ed7de5ef3b
[root@k8s-4 ~]# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qrouter-cbf00c95-b3ed-4f21-98fb-c7ed7de5ef3b
RTNETLINK answers: Invalid argument
qdhcp-5260ce27-36a4-4e93-9771-f9f90b94e057
RTNETLINK answers: Invalid argument
qdhcp-977a4d48-1613-457f-b99c-a31870f42858
RTNETLINK answers: Invalid argument
qdhcp-53ac44ee-5527-4a5e-a521-45ed2ed0231b
RTNETLINK answers: Invalid argument
qdhcp-2b37f37f-2c19-4759-81ca-1deac5860bfa
RTNETLINK answers: Invalid argument
qdhcp-04baff87-98c2-4ac0-b8e1-16b0ed692671

Revision history for this message
Matthew Taylor (matthew-taylor-f) wrote :

We're currently experiencing the same issue.

See notes: https://bugs.launchpad.net/kolla/+bug/1616268

Revision history for this message
Sam Yaple (s8m) wrote :

This is not a Neutron problem, nor can Neutron solve this problem. This a Docker/mount namespace issue.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for neutron because there has been no activity for 60 days.]

Changed in neutron:
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.