Activity log for bug #1795280

Date Who What changed Old value New value Message
2018-09-30 22:06:13 Tobias Urdin bug added bug
2018-09-30 22:06:31 Tobias Urdin description This is probably not neutron related, but need help with some input. On a 3.10 kernel on CentOS 7.5 by simply creating a network and deleting it properly terminates all processes, removes interfaces and deletes the network namespace. [root@controller ~]# uname -r 3.10.0-862.11.6.el7.x86_64 If running a later kernel like 4.18 there is some change that causes the namespace deletion to cause a OSError errno 16 device or resource busy. Before something like kernel 3.19 the netns filesystem was provided in proc but has since been moved to it's own nsfs, maybe this has something to do with it, but I haven't seen this issue on Ubuntu before. [root@controller ~]# mount | grep qdhcp proc on /run/netns/qdhcp-51e47959-9a2b-4372-a204-aff75de9bd01 type proc (rw,nosuid,nodev,noexec,relatime) proc on /run/netns/qdhcp-51e47959-9a2b-4372-a204-aff75de9bd01 type proc (rw,nosuid,nodev,noexec,relatime) [root@osc-network1-sto1-prod ~]# uname -r 4.18.8-1.el7.elrepo.x86_64 nsfs on /run/netns/qdhcp-1fb24615-fd9e-4804-aade-5668bb2cdecb type nsfs (rw,seclabel) nsfs on /run/netns/qdhcp-1fb24615-fd9e-4804-aade-5668bb2cdecb type nsfs (rw,seclabel) Perhaps some CentOS or RedHat person can shime in about this. Can reproduce this every single time: * Create network, it spawns dnsmasq, haproxy and the interfaces in a netns * Delete network, it will terminate all processes, delete interface but netns cannot be deleted and throws below error Seen on both queens and rocky fwiw 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent [req-28a9e37f-a2ca-4375-a3f0-8384711414dd - - - - -] Unable to disable dhcp for 1fb24615-fd9e-4804-aade-5668bb2cdecb.: OSError: [Errno 16] Device or resource busy 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 144, in call_driver 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 241, in disable 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent self._destroy_namespace_and_port() 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 255, in _destroy_namespace_and_port 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent ip_lib.delete_network_namespace(self.network.namespace) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1105, in delete_network_namespace 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent privileged.remove_netns(namespace, **kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent OSError: [Errno 16] Device or resource busy This is probably not neutron related, but need help with some input. On a 3.10 kernel on CentOS 7.5 by simply creating a network and deleting it properly terminates all processes, removes interfaces and deletes the network namespace. [root@controller ~]# uname -r 3.10.0-862.11.6.el7.x86_64 If running a later kernel like 4.18 there is some change that causes the namespace deletion to cause a OSError errno 16 device or resource busy. Before something like kernel 3.19 the netns filesystem was provided in proc but has since been moved to it's own nsfs, maybe this has something to do with it, but I haven't seen this issue on Ubuntu before. [root@controller ~]# mount | grep qdhcp proc on /run/netns/qdhcp-51e47959-9a2b-4372-a204-aff75de9bd01 type proc (rw,nosuid,nodev,noexec,relatime) proc on /run/netns/qdhcp-51e47959-9a2b-4372-a204-aff75de9bd01 type proc (rw,nosuid,nodev,noexec,relatime) [root@controller ~]# uname -r 4.18.8-1.el7.elrepo.x86_64 nsfs on /run/netns/qdhcp-1fb24615-fd9e-4804-aade-5668bb2cdecb type nsfs (rw,seclabel) nsfs on /run/netns/qdhcp-1fb24615-fd9e-4804-aade-5668bb2cdecb type nsfs (rw,seclabel) Perhaps some CentOS or RedHat person can shime in about this. Can reproduce this every single time: * Create network, it spawns dnsmasq, haproxy and the interfaces in a netns * Delete network, it will terminate all processes, delete interface but netns cannot be deleted and throws below error Seen on both queens and rocky fwiw 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent [req-28a9e37f-a2ca-4375-a3f0-8384711414dd - - - - -] Unable to disable dhcp for 1fb24615-fd9e-4804-aade-5668bb2cdecb.: OSError: [Errno 16] Device or resource busy 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent Traceback (most recent call last): 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp/agent.py", line 144, in call_driver 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent getattr(driver, action)(**action_kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 241, in disable 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent self._destroy_namespace_and_port() 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 255, in _destroy_namespace_and_port 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent ip_lib.delete_network_namespace(self.network.namespace) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 1105, in delete_network_namespace 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent privileged.remove_netns(namespace, **kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 207, in _wrap 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent return self.channel.remote_call(name, args, kwargs) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 202, in remote_call 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent raise exc_type(*result[2]) 2018-10-01 00:03:27.662 2093 ERROR neutron.agent.dhcp.agent OSError: [Errno 16] Device or resource busy
2018-10-01 09:43:37 Tobias Urdin bug watch added https://github.com/svinota/pyroute2/issues/535
2018-10-01 18:11:10 Brian Haley bug added subscriber Brian Haley
2018-10-01 18:11:18 Nate Johnston neutron: importance Undecided High