Layer 3 agent fails with rootwarp after reboot

[Gary Kotton]

ommand: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2bbbd7e8-7737-4185-be5a-90de2a13e94c', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: ip netns exec qrouter-2bbbd7e8-7737-4185-be5a-90de2a13e94c /sbin/iptables-save -t filter\n'
Stderr: ''
2012-09-16 04:58:52 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 175, in daemon_loop
    self.do_single_loop()
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 228, in do_single_loop
    self.process_router(ri)
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 281, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 381, in external_gateway_added
    ri.iptables_manager.apply()
  File "/usr/lib/python2.7/site-packages/quantum/agent/linux/iptables_manager.py", line 282, in apply
    root_helper=self.root_helper))
  File "/usr/lib/python2.7/site-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/13066

[dan wendlandt (danwent)]

This issue was reported in a separate bug (which I marked as a dup), so we need to figure out why the recursive behavior that Mark mentions in the review does not seem to be working properly.

[dan wendlandt (danwent)]

I'm seeing this on ubuntu as well. The first fix I tried by inspection was essentially the same as gary's, but it did not seem to solve the problem for me. I'll keep investigating.

[dan wendlandt (danwent)]

working on an alternate patch to this. will post.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/13093

[dan wendlandt (danwent)]

mark, gary, can you two take a look at the patch I posted? This is the fix I got by debugging the particular issue I was seeing, but I don't have my head wrapped around this enough to know if its the right patch.

[Gary Kotton (garyk)]

dan, it seems to solve the original problem that i was seeing (i'll match my patch as abandonded).
i am not encountering the following:

2012-09-17 02:03:18 INFO [quantum.agent.linux.iptables_manager] ===> ['ip', 'netns', 'exec', u'qrouter-2bbbd7e8-7737-4185-be5a-90de2a13e94c', '/sbin/iptables-save', '-t', 'filter']
2012-09-17 02:03:18 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 175, in daemon_loop
    self.do_single_loop()
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 228, in do_single_loop
    self.process_router(ri)
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 270, in process_router
    p['ip_cidr'], p['mac_address'])
  File "/usr/lib/python2.7/site-packages/quantum/agent/l3_agent.py", line 443, in internal_network_added
    ri.iptables_manager.apply()
  File "/usr/lib/python2.7/site-packages/quantum/agent/linux/iptables_manager.py", line 292, in apply
    root_helper=self.root_helper)
  File "/usr/lib/python2.7/site-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2bbbd7e8-7737-4185-be5a-90de2a13e94c', '/sbin/iptables-restore']
Exit code: 1

When I try and run the command manually I get the following:
[stack@(none) ~]$ sudo ip netns exec qrouter-2bbbd7e8-7737-4185-be5a-90de2a13e94c sbin/iptables-save -t filter
seting the network namespace failed: Invalid argument

This was a similar issue that I encountered with the deletion of the namespace. Basically it reaches a stage where we are unable to work with it. I am not sure if this issue is isolated to fedora or if it is related to the ovs with namspaces (i'll check with the linux bridge)

[Mark McClain (markmcclain)]

Does the namespace exist? Can manually run a bash shell in the namespace? (ip netns exec <ns> bash)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/13093
Committed: http://github.com/openstack/quantum/commit/a80d8b03421f0d920eaa336f3389ad2cad4f032b
Submitter: Jenkins
Branch: master

commit a80d8b03421f0d920eaa336f3389ad2cad4f032b
Author: Dan Wendlandt <email address hidden>
Date: Mon Sep 17 10:35:51 2012 -0700

    fix for nested rootwrap checks with 'ip netns exec'

    bug 1051525

    Change-Id: Ia52060afec1c573c8f11d658af88cea7e000d774

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/13274

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/13274
Committed: http://github.com/openstack/quantum/commit/55220838d0f7d95439a98e8a347bc84c1cc8c4a8
Submitter: Jenkins
Branch: milestone-proposed

commit 55220838d0f7d95439a98e8a347bc84c1cc8c4a8
Author: Dan Wendlandt <email address hidden>
Date: Mon Sep 17 10:35:51 2012 -0700

    fix for nested rootwrap checks with 'ip netns exec'

    bug 1051525

    Change-Id: Ia52060afec1c573c8f11d658af88cea7e000d774