neutron-ovn-metadata-agent uses network namespaces to separate the metadata services for individual networks. For each network it automatically creates or destroys an appropriate namespace.
If the metadata agent dies for reasons outside of its control (e.g. a SIGKILL) during the process of namespace destruction a broken namespace can be left over.
---
Background on pyroute2 namespace management:
Creating a network namespace works by:
1. Forking the process and doing everything in the new child
2. Ensuring /var/run/netns exists
3. Ensuring the file for the network namespace under /var/run/netns exists by creating a new empty file
4. calling `unshare` with `CLONE_NEWNET` to move the process to a new network namespace
5. Creating a bind mount from `/proc/self/ns/net` to the file under /var/run/netns
Deleting a network namespace works the other way around (but shorter):
1. Unmounting the previously created bind mount
2. Deleting the file for the network namespace
---
If the neutron-ovn-metadata-agent is killed between step 1 and 2 of deleting the network namespace then the namespace file will still be around, but not point to any namespace.
When `garbage_collect_namespace` tries to check if the namespace is empty it tries to enter the network namespace to dump all devices in there. This raises an exception as the namespace can no longer be entered.
neutron-ovn-metadata-agent then crashes and tries again next time, crashing again.
```
Traceback (most recent call last):,
File "/usr/local/bin/neutron-ovn-metadata-agent", line 8, in <module>,
sys.exit(main()),
File "/usr/local/lib/python3.9/site-packages/neutron/cmd/eventlet/agents/ovn_metadata.py", line 24, in main,
metadata_agent.main(),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/ovn/metadata_agent.py", line 41, in main,
agt.start(),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/ovn/metadata/agent.py", line 277, in start,
self.sync(),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/ovn/metadata/agent.py", line 61, in wrapped,
return f(*args, **kwargs),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/ovn/metadata/agent.py", line 349, in sync,
self.teardown_datapath(self._get_datapath_name(ns)),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/ovn/metadata/agent.py", line 400, in teardown_datapath,
ip.garbage_collect_namespace(),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 268, in garbage_collect_namespace,
if self.namespace_is_empty():,
File "/usr/local/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 263, in namespace_is_empty,
return not self.get_devices(),
File "/usr/local/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 180, in get_devices,
devices = privileged.get_device_names(self.namespace),
File "/usr/local/lib/python3.9/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 609, in get_device_names,
in get_link_devices(namespace, **kwargs)],
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 333, in wrapped_f,
return self(f, *args, **kw),
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__,
do = self.iter(retry_state=retry_state),
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 360, in iter,
return fut.result(),
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result,
return self.__get_result(),
File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result,
raise self._exception,
File "/usr/local/lib/python3.9/site-packages/tenacity/__init__.py", line 426, in __call__,
result = fn(*args, **kwargs),
File "/usr/local/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap,
return self.channel.remote_call(name, args, kwargs,,
File "/usr/local/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call,
raise exc_type(*result[2]),
OSError: [Errno 22] failed to open netns
```
Versions: afaik affects all versions
Reproduction: best by creating a empty file with the name `/var/run/netns/ovnmeta-<some-uuid>` and restarting the neutron-ovn-metadata-agent. Otherwise a breakpoint or a good timed kill command
Fix proposed to branch: master
Review: https:/