rootwrap error in devstack w/quantum from quantum-dhcp

Bug #1080846 reported by Robert Collins
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Gary Kotton
Grizzly
Fix Released
High
Gary Kotton

Bug Description

2012-11-19 20:40:04 7218 ERROR quantum.agent.dhcp_agent [-] Unable to reload_allocations dhcp.
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent Traceback (most recent call last):
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/dhcp_agent.py", line 90, in call_driver
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent getattr(driver, action)()
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/dhcp.py", line 278, in reload_allocations
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent ip_wrapper.netns.execute(cmd)
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 351, in execute
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent check_exit_code=check_exit_code)
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/utils.py", line 59, in execute
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent raise RuntimeError(m)
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent RuntimeError:
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent Command: ['sudo', '/opt/stack/quantum/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-a10792e8-3787-471b-b55b-dbe1ba4e95a3', 'kill', '-HUP', '8729']
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent Exit code: 99
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent Stdout: 'Unauthorized command: ip netns exec qdhcp-a10792e8-3787-471b-b55b-dbe1ba4e95a3 kill -HUP 8729\n'
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent Stderr: ''
2012-11-19 20:40:04 7218 TRACE quantum.agent.dhcp_agent

Tags: l3-ipam-dhcp
Revision history for this message
Robert Collins (lifeless) wrote :

emilien suggested using 'sudo' as the root_helper in the quantum-dhcp.ini file.

That permits the command to work :

2012-11-19 20:50:53 14448 DEBUG quantum.agent.linux.utils [-] Running command: sudo ip netns exec qdhcp-a10792e8-3787-471b-b55b-dbe1ba4e95a3 ip -o link show tap8181df00-d9 execute /opt/stack/quantum/quantum/agent/linux/utils.py:41
2012-11-19 20:50:53 14448 DEBUG quantum.agent.linux.utils [-]
Command: ['sudo', 'ip', 'netns', 'exec', 'qdhcp-a10792e8-3787-471b-b55b-dbe1ba4e95a3', 'ip', '-o', 'link', 'show', 'tap8181df00-d9']
Exit code: 0

Revision history for this message
Robert Collins (lifeless) wrote :

I don't know if it is relevant, but devstack has installed the openvswitch rootwrap filter as 'stack:stack' rather than 'root:root', perhaps rootwrap is refusing to use the file as its owned by non-root?

Revision history for this message
Gary Kotton (garyk) wrote :

Hi,
When did you last pull in the Quantum code. I did this with the latest code this morning and it works. Please see below:

2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.utils [-] Running command: sudo /opt/stack/quantum/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-7b761701-e01e-4c16-a6eb-aba54bd0b9f0 kill -HUP 6700 execute /opt/stack/quantum/quantum/agent/linux/utils.py:41
2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.utils [-]
Command: ['sudo', '/opt/stack/quantum/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-7b761701-e01e-4c16-a6eb-aba54bd0b9f0', 'kill', '-HUP', '6700']
Exit code: 0
Stdout: ''
Stderr: '' execute /opt/stack/quantum/quantum/agent/linux/utils.py:57
2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.dhcp [-] Reloading allocations for network: 7b761701-e01e-4c16-a6eb-aba54bd0b9f0 reload_allocations /opt/stack/quantum/quantum/agent/linux/dhcp.py:281

Thanks
Gary

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 1080846] Re: rootwrap error in devstack w/quantum from quantum-dhcp

On Tue, Nov 20, 2012 at 7:56 PM, Gary Kotton <email address hidden> wrote:
> Hi,
> When did you last pull in the Quantum code. I did this with the latest code this morning and it works. Please see below:
>
> 2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.utils [-] Running command: sudo /opt/stack/quantum/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-7b761701-e01e-4c16-a6eb-aba54bd0b9f0 kill -HUP 6700 execute /opt/stack/quantum/quantum/agent/linux/utils.py:41
> 2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.utils [-]
> Command: ['sudo', '/opt/stack/quantum/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-7b761701-e01e-4c16-a6eb-aba54bd0b9f0', 'kill', '-HUP', '6700']
> Exit code: 0
> Stdout: ''
> Stderr: '' execute /opt/stack/quantum/quantum/agent/linux/utils.py:57
> 2012-11-20 08:59:54 5793 DEBUG quantum.agent.linux.dhcp [-] Reloading allocations for network: 7b761701-e01e-4c16-a6eb-aba54bd0b9f0 reload_allocations /opt/stack/quantum/quantum/agent/linux/dhcp.py:281
>
> Thanks
> Gary
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1080846
>
> Title:
> rootwrap error in devstack w/quantum from quantum-dhcp
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/quantum/+bug/1080846/+subscriptions

Fresh machine right before reporting the error.

--
Robert Collins <email address hidden>
Distinguished Technologist
HP Cloud Services

dan wendlandt (danwent)
Changed in quantum:
importance: Undecided → High
Revision history for this message
dan wendlandt (danwent) wrote :

I have tested this on a clean install of 12.04, and do not see such errors in my logs, and can confirm that the VM properly gets a DHCP address.

all of my /etc/quantum/rootwrap.conf and /etc/quantum/rootwrap.d/ files are owned by stack:stack.

my only thinking is that I think rootwrap checks if a pid is valid before allowing a KILL on it, in which case perhaps the above pid is not valid.

Changed in quantum:
status: New → Incomplete
Revision history for this message
Robert Collins (lifeless) wrote :

So, fresh quantal based devstack today, reproduced this again.

stack@ubuntu:~/devstack$ ls -l /etc/quantum/rootwrap.conf
-rw-r--r-- 1 root root 178 Dec 20 07:31 /etc/quantum/rootwrap.conf
stack@ubuntu:~/devstack$ ls -l /etc/quantum/rootwrap.conf /etc/quantum/rootwrap.d/
-rw-r--r-- 1 root root 178 Dec 20 07:31 /etc/quantum/rootwrap.conf

/etc/quantum/rootwrap.d/:
total 32
-rw-r--r-- 1 root root 483 Dec 20 07:09 debug.filters
-rw-r--r-- 1 root root 1155 Dec 20 07:09 dhcp.filters
-rw-r--r-- 1 root root 719 Dec 20 07:09 iptables-firewall.filters
-rw-r--r-- 1 root root 1521 Dec 20 07:09 l3.filters
-rw-r--r-- 1 root root 623 Dec 20 07:09 linuxbridge-plugin.filters
-rw-r--r-- 1 root root 485 Dec 20 07:09 nec-plugin.filters
-rw-r--r-- 1 root root 1043 Dec 20 07:09 openvswitch-plugin.filters
-rw-r--r-- 1 root root 816 Dec 20 07:09 ryu-plugin.filters

I'll try changing them to stack:stack

Revision history for this message
Robert Collins (lifeless) wrote :

(though, if they are owned by stack, there is no root wrap protection anymore, may as well just use plain sudo - rootwraps reason for existence is to gate access to root such that openstack services can't bypass it :))

Revision history for this message
Robert Collins (lifeless) wrote :

File owner made no difference.

Changed in quantum:
status: Incomplete → New
Revision history for this message
Robert Collins (lifeless) wrote :

It does appear to be the PID:

 sudo /usr/local/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-788fd34d-6201-49c6-bb0d-928eec877607 kill -HUP 22774
Unauthorized command: ip netns exec qdhcp-788fd34d-6201-49c6-bb0d-928eec877607 kill -HUP 22774
stack@ubuntu:~/devstack$ ps fax | grep 22774
21710 pts/1 S+ 0:00 | \_ grep --color=auto 22774
stack@ubuntu:~/devstack$ ps fax | grep dhcp
  818 ? Ss 0:00 dhclient -1 -v -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
21930 pts/1 S+ 0:00 | \_ grep --color=auto dhcp
22447 pts/9 S+ 0:01 | \_ python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini
28009 ? S 0:01 dnsmasq --conf-file= --port=0 --enable-tftp --tftp-root=/tftpboot --dhcp-boot=pxelinux.0 --bind-interfaces --pid-file=/var/run/dnsmasq.pid --interface=eth1 --dhcp-range=192.0.2.65,192.0.2.127,255.255.255.0
stack@ubuntu:~/devstack$ sudo /usr/local/bin/quantum-rootwrap /etc/quantum/rootwrap.conf ip netns exec qdhcp-788fd34d-6201-49c6-bb0d-928eec877607 kill -HUP 28009

Revision history for this message
Aaron Rosen (arosen) wrote :

What's the content of: /etc/sudoers.d/quantum-rootwrap ?

It should be:
arosen ALL=(root) NOPASSWD: /usr/local/bin/quantum-rootwrap /etc/quantum/rootwrap.conf *

And the quantum-dhcp-agent should be running as the username at the start of that line (in this case arosen).

Revision history for this message
Robert Collins (lifeless) wrote :

stack@ubuntu:~$ sudo less /etc/sudoers.d/quantum-rootwrap
stack ALL=(root) NOPASSWD: /usr/local/bin/quantum-rootwrap
/etc/quantum/rootwrap.conf *
/etc/sudoers.d/quantum-rootwrap (END)

Revision history for this message
Aaron Rosen (arosen) wrote :

Great, now when the quantum-dhcp-agent is running if you run:

ps -eaf | grep quantum-dhcp-agent

It should return:

arosen 1553 1493 0 Jan03 pts/23 00:04:13 python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini

But in your case arosen should say stack. Does it?

--- Also in my setup: /etc/quantum/rootwrap.d is all owned by root. I'm pretty sure that needs to be owned by root to work otherwise your user could add commands to that which would make the rootwrap useless.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Hi Aaron. I'm seeing the exact same thing, and I do have the same ps -eaf output:

$ ps -eaf | grep quantum-dhcp-agent
stack 1265 1144 0 21:11 pts/10 00:00:02 python /opt/stack/quantum/bin/quantum-dhcp-agent --config-file /etc/quantum/quantum.conf --config-file=/etc/quantum/dhcp_agent.ini

error in dhcp agent log from screen:

Stdout: 'Unauthorized command: ip netns exec qdhcp-0918f496-40f3-4810-8c01-afb3bc0753bf kill -HUP 2070\n'
Stderr: '' execute /opt/stack/quantum/quantum/agent/linux/utils.py:58
2013-01-14 22:02:19.589 1265 ERROR quantum.agent.dhcp_agent [-] Unable to reload_allocations dhcp.
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent Traceback (most recent call last):
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/dhcp_agent.py", line 89, in call_driver
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent getattr(driver, action)()
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/dhcp.py", line 278, in reload_allocations
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent ip_wrapper.netns.execute(cmd)
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/ip_lib.py", line 391, in execute
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent check_exit_code=check_exit_code)
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent File "/opt/stack/quantum/quantum/agent/linux/utils.py", line 60, in execute
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent raise RuntimeError(m)
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent RuntimeError:
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent Command: ['sudo', '/usr/local/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-0918f496-40f3-4810-8c01-afb3bc0753bf', 'kill', '-HUP', '2070']
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent Exit code: 99
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent Stdout: 'Unauthorized command: ip netns exec qdhcp-0918f496-40f3-4810-8c01-afb3bc0753bf kill -HUP 2070\n'
2013-01-14 22:02:19.589 1265 TRACE quantum.agent.dhcp_agent Stderr: ''

Changed in quantum:
status: New → Confirmed
tags: added: l3-ipam-dhcp
dan wendlandt (danwent)
Changed in quantum:
assignee: nobody → dan wendlandt (danwent)
Revision history for this message
Thierry Carrez (ttx) wrote :

Looks like everything works fine, except dnsmasq no longer lives at the expected PID. Since rootwrap's KillFilter looks up the PID to check that you indeed affect a dnsmasq process, if that PID lookup fails, it rejects the whole command...

Revision history for this message
Thierry Carrez (ttx) wrote :

I suspect this is actually a Quantum equivalent of https://bugs.launchpad.net/nova/+bug/1010275

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/28037

Changed in quantum:
assignee: dan wendlandt (danwent) → Gary Kotton (garyk)
status: Confirmed → In Progress
Gary Kotton (garyk)
tags: added: grizzly-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (master)

Reviewed: https://review.openstack.org/28037
Committed: http://github.com/openstack/quantum/commit/9274095b4af63de7224b524e482872a78e027a7b
Submitter: Jenkins
Branch: master

commit 9274095b4af63de7224b524e482872a78e027a7b
Author: Gary Kotton <email address hidden>
Date: Thu May 2 11:35:58 2013 +0000

    Do not attempt to kill already-dead dnsmasq

    Fixes bug 1080846

    The fix is following comments by Thiery Carrez (ttx) on the bug.

    Change-Id: If4f6baad4212c23845c46703140e15f1ffcfe558

Changed in quantum:
status: In Progress → Fix Committed
Changed in quantum:
milestone: none → havana-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/grizzly)

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/28146

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (stable/grizzly)

Reviewed: https://review.openstack.org/28146
Committed: http://github.com/openstack/quantum/commit/7addf4152497821ab646582ad4c706d35a2c3088
Submitter: Jenkins
Branch: stable/grizzly

commit 7addf4152497821ab646582ad4c706d35a2c3088
Author: Gary Kotton <email address hidden>
Date: Thu May 2 11:35:58 2013 +0000

    Do not attempt to kill already-dead dnsmasq

    Fixes bug 1080846

    The fix is following comments by Thiery Carrez (ttx) on the bug.

    Change-Id: If4f6baad4212c23845c46703140e15f1ffcfe558

tags: added: in-stable-grizzly
Gary Kotton (garyk)
tags: removed: grizzly-backport-potential
Thierry Carrez (ttx)
Changed in quantum:
status: Fix Committed → Fix Released
Alan Pevec (apevec)
tags: removed: in-stable-grizzly
Thierry Carrez (ttx)
Changed in neutron:
milestone: havana-1 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.