Comment 3 for bug 1864856

This is reproducible on centos7/train (Dincer case):
need to reconfigure and (extra) restart l3 agent container.

The issue is that on 2nd restart container has invalid references to netns inside itself - instead of proc mounts, it has a bunch of regular files which are unusable for l3 agent.

Logs:

2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent [-] Error while initializing router 87c3c36f-1038-45d4-afc0-c9c109239b66: ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: setting the network namespace "qrouter-87c3c36f-1038-45d4-afc0-c9c109239b66" failed: Inv
alid argument
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 485, in _router_added
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent ri.initialize(self.process_monitor)
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 167, in initialize
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent self.router_namespace.create()
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 97, in create
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent ip_wrapper.netns.execute(cmd)
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 713, in execute
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent run_as_root=run_as_root)
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 147, in execute
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent returncode=returncode)
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent ProcessExecutionError: Exit code: 1; Stdin: ; Stdout: ; Stderr: setting the network namespace "qrouter-87c3c36f-1038-45d4-afc0-c9c109239b66" failed: Invalid argument
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent
2020-02-27 12:20:58.626 17 ERROR neutron.agent.l3.agent

2020-02-27 12:20:58.644 17 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for 87c3c36f-1038-45d4-afc0-c9c109239b66, action 3
2020-02-27 12:20:58.644 17 WARNING neutron.agent.l3.agent [-] Info for router 87c3c36f-1038-45d4-afc0-c9c109239b66 was not found. Performing router cleanup
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent [-] Error while deleting router 87c3c36f-1038-45d4-afc0-c9c109239b66: OSError: [Errno 22] failed to open netns
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 506, in _safe_router_removed
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent self._router_removed(ri, router_id)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 529, in _router_removed
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent self.namespaces_manager.ensure_router_cleanup(router_id)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/namespace_manager.py", line 129, in ensure_router_cleanup
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent self._cleanup(ns_prefix, ns_id)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/namespace_manager.py", line 143, in _cleanup
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent ns.delete()
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 72, in wrapped
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent return f(self, *args, **kwargs)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 136, in delete
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent for d in ns_ip.get_devices():
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 167, in get_devices
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent devices = privileged.get_device_names(self.namespace)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 582, in get_device_names
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent in get_link_devices(namespace, **kwargs)]
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/neutron/privileged/agent/linux/ip_lib.py", line 53, in sync_inner
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent return input_func(*args, **kwargs)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs)
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 204, in remote_call
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent OSError: [Errno 22] failed to open netns
2020-02-27 12:20:58.657 17 ERROR neutron.agent.l3.agent