Failed to delete router namespace when router-delete

[Akihiro Motoki]

Failed to delete router namespace when router-delete.

After stack.sh with q-l3 enabled, invoke the following command sequence.
(1) quantum router-gateway-clear router1
(2) quantum router-interface-delete router1 <subnet-id>
(3) quantum router-delete router1

When deleting router1, L3 agent output the following logs.
In my devtack environment it always happens (5/5)

I can not identify the reason so far. Can anyone investigate it?

quantum: c0473e1 Merge "Add enable_tunneling openvswitch configuration variable"
devstack: 3ceb380 Merge "Devstack support for Quantum L3 agent"

-----
2012-09-11 16:02:33 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns exec qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0 ip -o link list
2012-09-11 16:02:33 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'exec', 'qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0', 'ip', '-o', 'link', 'list']
Exit code: 0
Stdout: '344: lo: <LOOPBACK> mtu 16436 qdisc noop state DOWN \\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n'
Stderr: ''
2012-09-11 16:02:33 DEBUG [quantum.agent.linux.utils] Running command: sudo ip netns delete qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0
2012-09-11 16:02:33 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', 'ip', 'netns', 'delete', 'qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0']
Exit code: 1
Stdout: ''
Stderr: 'Cannot remove /var/run/netns/qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0: Device or resource busy\n'
2012-09-11 16:02:33 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/opt/stack/quantum/quantum/agent/l3_agent.py", line 174, in daemon_loop
    self.do_single_loop()
  File "/opt/stack/quantum/quantum/agent/l3_agent.py", line 233, in do_single_loop
    self._destroy_router_namespace(ri.ns_name())
  File "/opt/stack/quantum/quantum/agent/l3_agent.py", line 157, in _destroy_router_namespace
    ns_ip.netns.delete(namespace)
  File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 339, in delete
    self._as_root('delete', name, use_root_namespace=True)
  File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 158, in _as_root
    kwargs.get('use_root_namespace', False))
  File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 45, in _as_root
    namespace)
  File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 56, in _execute
    root_helper=root_helper)
  File "/opt/stack/quantum/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'ip', 'netns', 'delete', 'qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0']
Exit code: 1
Stdout: ''
Stderr: 'Cannot remove /var/run/netns/qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0: Device or resource busy\n'
2012-09-11 16:02:36 DEBUG [quantumclient.client] REQ: curl -i http://10.56.51.210:9696//v2.0/networks.json?router%3Aexternal=True -X GET -H "User-Agent: python-quantumclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: a9767b94d28e4f52b5054d993951f59a"
------

After the above, the following command also fails:
-----
$ sudo ip netns exec qrouter-a3b31791-2f0a-4e8c-812e-fd3457119db0 ip -o link
seting the network namespace failed: Invalid argument
-----

[Salvatore Orlando (salvatore-orlando)]

It looks like this is the same issue Gary and Mark are investigating (and hopefully sorted by now)

[Akihiro Motoki (amotoki)]

Hi Salvatore,

Hmm... It seems not to be related to bug 1048968.
I tested with quantum after Gary's patch has been merged, but it still exists.

If my memory is correct, you mentioned that "sudo ip netns exec <namespace> ip -o link" sometimes fails in the review of netns clean utitlity.

[Gary Kotton (garyk)]

Hii Akihiro,
As far as I have tested the patch https://review.openstack.org/#/c/12691/ should address this.
It does the following:
1. ensures that the namepaces are not deleted (unless I have missed out one of the places)
2. if a route exists and it is created again then the agent will not fail.
Thanks
Gary

[Gary Kotton (garyk)]

I think that we shoul;d mark this a duplicate of 1048108

[Salvatore Orlando (salvatore-orlando)]

I was actually referring to the bug mentioned bty gary too: https://bugs.launchpad.net/quantum/+bug/1048108

[Gary Kotton (garyk)]

i was unable to reproduce this with the patch https://review.openstack.org/#/c/12691/.
i am going to duplicrate

[Akihiro Motoki (amotoki)]

Sary, Salvatore,
Thanks I got it. It is a duplicate.