deleting router run into race condition

[Bernhard]

After deleting a router the logfiles of both network nodes are filled up with " RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-3767"
After i have restarted the openstack services on the network nodes, no new entries

Reproduceable: yes

Steps:
* add router via CLI or dashboard
* delete router via CLI or dashboard
* logfiles grow up

Openstack version: mitaka ( this error occured on liberty too ! )

OS: Centos 7, latest updates

Installed Packages on nerwork nodes
openstack-neutron-vpnaas-8.0.0-1.el7.noarch
openstack-neutron-common-8.1.2-1.el7.noarch
openstack-neutron-metering-agent-8.1.2-1.el7.noarch
python-neutronclient-4.1.1-2.el7.noarch
python-neutron-8.1.2-1.el7.noarch
python-neutron-fwaas-8.0.0-3.el7.noarch
openstack-neutron-ml2-8.1.2-1.el7.noarch
openstack-neutron-bgp-dragent-8.1.2-1.el7.noarch
python-neutron-vpnaas-8.0.0-1.el7.noarch
openstack-neutron-openvswitch-8.1.2-1.el7.noarch
openstack-neutron-8.1.2-1.el7.noarch
python-neutron-lib-0.0.2-1.el7.noarch
openstack-neutron-fwaas-8.0.0-3.el7.noarch

Logfile network node:
2.770 44778 DEBUG neutron.agent.linux.ra [-] radvd disabled for router 37678766-597a-4e33-b83a-65142ca2ced8 disable /usr/lib/python2.7/site-packages/neutron/agent/linux/ra.py:190
2016-07-27 09:10:02.770 44778 DEBUG neutron.agent.linux.utils [-] Running command (rootwrap daemon): ['ip', 'netns', 'exec', 'qrouter-37678766-597a-4e33-b83a-65142ca2ced8', 'find', '/sys/class/net', '-maxdepth', '1', '-type', 'l', '-printf', '%f '] execute_rootwrap_daemon /usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py:100
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent [-] Error while deleting router 37678766-597a-4e33-b83a-65142ca2ced8
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 359, in _safe_router_removed
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent self._router_removed(router_id)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py", line 377, in _router_removed
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent ri.delete(self)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/ha_router.py", line 380, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent super(HaRouter, self).delete(agent)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/router_info.py", line 349, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent self.router_namespace.delete()
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/l3/namespaces.py", line 100, in delete
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent for d in ns_ip.get_devices(exclude_loopback=True):
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 130, in get_devices
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent log_fail_as_error=self.log_fail_as_error
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 140, in execute
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent raise RuntimeError(msg)
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent RuntimeError: Exit code: 1; Stdin: ; Stdout: ; Stderr: Cannot open network namespace "qrouter-37678766-597a-4e33-b83a-65142ca2ced8": No such file or directory
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent
2016-07-27 09:10:02.773 44778 ERROR neutron.agent.l3.agent

Attached logfiles of control node and both network nodes.
At 09:09:00 -> added router
At 09:10:00 -> deleted router

[Bernhard (b-krieger)]

As i see this reported bug is same as https://bugs.launchpad.net/neutron/+bug/1605546

[Bernhard (b-krieger)]

After i have applied the patch i get still the exceptions

from vpn-agent.log:

2016-08-01 09:41:26.377 53157 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2016-08-01 09:41:26.380 53157 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.003s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-08-01 09:41:26.429 53157 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" acquired by "neutron.agent.linux.external_process._check_child_processes" :: waited 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:270
2016-08-01 09:41:26.429 53157 DEBUG oslo_concurrency.lockutils [-] Lock "_check_child_processes" released by "neutron.agent.linux.external_process._check_child_processes" :: held 0.000s inner /usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py:282
2016-08-01 09:41:26.523 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 743c08e13d104d73b6c122fbbb1becae exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-08-01 09:41:26.538 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: 743c08e13d104d73b6c122fbbb1becae __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-08-01 09:41:48.585 53157 DEBUG oslo_service.periodic_task [req-c87d0ab2-94d9-4fcb-967a-9640ed3c1f59 - - - - -] Running periodic task VPNAgent.periodic_sync_routers_task run_periodic_tasks /usr/lib/python2.7/site-packages/oslo_service/periodic_task.py:215
2016-08-01 09:41:56.519 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: f7123d03a9744005b710b144545de115 exchange 'neutron' topic 'q-reports-plugin' _send /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:454
2016-08-01 09:41:56.533 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] received reply msg_id: f7123d03a9744005b710b144545de115 __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:302
2016-08-01 09:42:07.823 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] received message msg_id: None reply to None __call__ /usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py:201
2016-08-01 09:42:07.825 53157 DEBUG neutron.agent.l3.agent [req-8bf4976f-a606-42a7-b60d-7baebe276155 4afea4ec82774aa3bfc4dab741c29934 352cd4ef40c04b058ba40316140c5a29 - - -] Got routers updated notification :[u'a9806b97-0850-4f40-8c73-c1fac666d21b'] routers_updated /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:392
2016-08-01 09:42:07.825 53157 DEBUG neutron.agent.l3.agent [-] Starting router update for a9806b97-0850-4f40-8c73-c1fac666d21b, action None, priority 0 _process_router_update /usr/lib/python2.7/site-packages/neutron/agent/l3/agent.py:467
2016-08-01 09:42:07.826 53157 DEBUG oslo_messaging._drivers.amqpdriver [-] CALL msg_id: 882a5673e9a84c33be219d327487a427 exchan...