Neutron router not working with latest iproute2 package included in CentOS-7.2-1511

[Andrew Poltavchenko]

Seems that something has been changed in the new iproute version and now attempts to add more than one interface to router cause errors posted at the bottom. This affects neutron-l3-agent on CentOS-7.2-1511 and possible on RedHat (cannot check this).

Quick solution is to simple downgrade the package:
# wget http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/iproute-3.10.0-21.el7.x86_64.rpm
# yum -y downgrade ./iproute-3.10.0-21.el7.x86_64.rpm

Part of the /var/log/neutron/l3-agent.log:

2015-12-24 01:35:05.794 6343 ERROR neutron.agent.linux.utils [-]
Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists

2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info [-]
Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
Exit code: 1
Stdin:
Stdout:
Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Traceback (most recent call last):
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info return func(*args, **kwargs)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._process_internal_ports(agent.pd)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.internal_network_added(p)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info INTERNAL_DEV_PREFIX)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info prefix=prefix)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info bridge, namespace, prefix)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info namespace2=namespace)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self.ensure_namespace(namespace2)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info ip = self.netns.add(name)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info self._as_root([], ('add', name), use_root_namespace=True)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info use_root_namespace=use_root_namespace)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info log_fail_as_error=self.log_fail_as_error)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info log_fail_as_error=log_fail_as_error)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info raise RuntimeError(m)
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info RuntimeError:
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Exit code: 1
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stdin:
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stdout:
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info
2015-12-24 01:35:05.794 6343 ERROR neutron.agent.l3.router_info
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent [-] Failed to process compatible router 'ec62eace-0415-49b5-9c26-dca1677ba85a'
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 498, in _process_router_update
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 443, in _process_router_if_compatible
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 457, in _process_updated_router
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent ri.process(self)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 359, in call
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self.logger(e)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 195, in __exit__
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent six.reraise(self.type_, self.value, self.tb)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/common/utils.py", line 356, in call
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 692, in process
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self._process_internal_ports(agent.pd)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 396, in _process_internal_ports
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self.internal_network_added(p)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 328, in internal_network_added
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent INTERNAL_DEV_PREFIX)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 303, in _internal_network_added
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent prefix=prefix)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 252, in plug
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent bridge, namespace, prefix)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/interface.py", line 483, in plug_new
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent namespace2=namespace)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 144, in add_veth
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self.ensure_namespace(namespace2)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 163, in ensure_namespace
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent ip = self.netns.add(name)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 793, in add
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent self._as_root([], ('add', name), use_root_namespace=True)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 280, in _as_root
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent use_root_namespace=use_root_namespace)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 80, in _as_root
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 89, in _execute
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent log_fail_as_error=log_fail_as_error)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 159, in execute
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent raise RuntimeError(m)
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent RuntimeError:
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Command: ['ip', 'netns', 'add', u'qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a']
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Exit code: 1
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stdin:
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stdout:
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent Stderr: Cannot create namespace file "/var/run/netns/qrouter-ec62eace-0415-49b5-9c26-dca1677ba85a": File exists
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent
2015-12-24 01:35:05.796 6343 ERROR neutron.agent.l3.agent

[Assaf Muller (amuller)]

Did you try this patch? https://review.openstack.org/#/q/I0d3fc4262284172f5ad31e4f2f78ae1fb33b4228,n,z

[Andrew Poltavchenko (andreipoltavchenko)]

Not yet, will test it today and reply you back.

[Wenxin Wang (stieizc-33)]

I'm experiencing the same thing. This is preventing the l3_agent from creating the router namespace. I can confirm this bug and the fix on CentOS 7.2.1511.

[Andrew Poltavchenko (andreipoltavchenko)]

Assaf, thank you for the patch: it is working for me (at least tested with router ports adding).

[Assaf Muller (amuller)]

Closing as a duplicate of bug 1497309.

[jost (jost-rakovec)]

Can someone explain how to apply patch. thanks