neutron-dhcp-agent is unable to set network namespaces

[Dilip Renkila]

Hi all,

I am running neutron-dhcp-agents on all my controller nodes. I figured out that neutron-dhcp-agent has a problem to work with namespaces. It is failing continuously. I have deployed openstack by using openstack ansible and all my networking agents run on the hosts not on containers.

root@ctrl1:~# uname -a
Linux ctrl1 4.4.0-130-generic #156-Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
You have new mail in /var/mail/root

root@ctrl1:~# /openstack/venvs/neutron-18.0.0.0b3/bin/neutron-dhcp-agent --version
neutron-dhcp-agent 12.0.0.0rc2.dev241

root@ctrl1:~# ls -la /var/run/netns
total 0
drwxrwxrwx 2 root root 100 aug 8 11:59 .
drwxr-xr-x 44 root root 1520 aug 8 16:11 ..
---------- 1 root root 0 aug 8 11:59 qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
---------- 1 root root 0 aug 7 11:28 qrouter-3755419c-2a20-4ed0-9012-af24863f2447
---------- 1 root root 0 aug 7 11:28 snat-3755419c-2a20-4ed0-9012-af24863f2447

root@ctrl3:~# ip netns
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
RTNETLINK answers: Invalid argument
qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127
RTNETLINK answers: Invalid argument
qrouter-3755419c-2a20-4ed0-9012-af24863f2447
RTNETLINK answers: Invalid argument
snat-3755419c-2a20-4ed0-9012-af24863f2447

root@ctrl1:~# tail -f /var/log/neutron/neutron-dhcp-agent.log

2018-08-08 16:06:32.408 1039014 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument

2018-08-08 16:06:32.409 1039014 WARNING neutron.agent.linux.ip_lib [-] Setting ['sysctl', '-w', 'net.ipv6.conf.default.accept_ra=0'] in namespace qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127 failed: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
setting the network namespace "qdhcp-b43d85c9-575d-40f5-8960-2ba26620e127" failed: Invalid argument
.: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: RTNETLINK answers: Invalid argument
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent [-] Unable to enable dhcp for b43d85c9-575d-40f5-8960-2ba26620e127.: OSError: [Errno 13] Permission denied
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent Traceback (most recent call last):
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 144, in call_driver
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 217, in enable
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent interface_name = self.device_manager.setup(self.network)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 1438, in setup
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent namespace=network.namespace):
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1111, in ensure_device_is_ready
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent if not dev.link.address:
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 545, in address
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent return self.attributes.get('link/ether')
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 574, in attributes
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent self._parent.namespace)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs)
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent File "/openstack/venvs/neutron-18.0.0.0b3/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2])
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent OSError: [Errno 13] Permission denied
2018-08-08 16:06:32.419 1039014 ERROR neutron.agent.dhcp.agent
2018-08-08 16:06:32.420 1039014 INFO neutron.agent.dhcp.agent [-] Finished network b43d85c9-575d-40f5-8960-2ba26620e127 dhcp configuration

[Brian Haley (brian-haley)]

I've seen errors like this when using containers, or with nested namespaces. Is the dhcp-agent getting started inside a namespace? I have not done a deployment with openstack ansible directly to know what else you can check.

[Dilip Renkila (dilip-renkila278)]

DHCP agent is not started inside the namespace. But i am not running agents inside containers.

[Bernard Cafarelli (bcafarel)]

This may be better redirected to openstack-ansible? In other deployments the DHCP agent handles namespaces fine.

Not an expert in openstack-ansible, but you may have a collision with environment set in the containers (like namespaces). Can you describe how you deployed this setup? which parts were manually installed, which parts come from openstack-ansible, which set the configuration, etc

[Launchpad Janitor (janitor)]

[Expired for neutron because there has been no activity for 60 days.]