dnsmasq - Stderr: 'Option "-no-hosts" is unknown, try "ip -help".\n'

[Endre Karlson]

root@os-svc02:~# quantum-dhcp-agent --config-file /etc/quantum/dhcp_agent.ini --config-file /etc/quantum/quantum.conf --debug
2012-09-24 11:05:56 DEBUG [amqplib] Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.8.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
2012-09-24 11:05:56 DEBUG [amqplib] Open OK! known_hosts []
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open
2012-09-24 11:05:56 INFO [quantum.openstack.common.rpc.common] Connected to AMQP server on mq.os.lan:5672
2012-09-24 11:05:56 INFO [quantum.agent.dhcp_agent] Synchronizing state
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is d2b03a27f9f64016ac41e5831be0decf
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] Pool creating new connection
2012-09-24 11:05:56 DEBUG [amqplib] Start from server, version: 8.0, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2012 VMware, Inc.', u'capabilities': {}, u'platform': u'Erlang/OTP', u'version': u'2.8.6'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
2012-09-24 11:05:56 DEBUG [amqplib] Open OK! known_hosts []
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open
2012-09-24 11:05:56 INFO [quantum.openstack.common.rpc.common] Connected to AMQP server on mq.os.lan:5672
2012-09-24 11:05:56 DEBUG [amqplib] Closed channel #1
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is a08cc1b5133f4d619db4af963635911c
2012-09-24 11:05:56 DEBUG [amqplib] Closed channel #1
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is 2b9f1ef565824cbe9bd53622ec79fede
2012-09-24 11:05:56 DEBUG [amqplib] Closed channel #1
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils] Running command: ip -o link show tapbd847180-7f
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils]
Command: ['ip', '-o', 'link', 'show', 'tapbd847180-7f']
Exit code: 0
Stdout: '18: tapbd847180-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \\ link/ether fa:16:3e:34:86:5d brd ff:ff:ff:ff:ff:ff\n'
Stderr: ''
2012-09-24 11:05:56 DEBUG [quantum.agent.dhcp_agent] Reusing existing device: tapbd847180-7f.
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils] Running command: ip addr show tapbd847180-7f permanent scope global
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils]
Command: ['ip', 'addr', 'show', 'tapbd847180-7f', 'permanent', 'scope', 'global']
Exit code: 0
Stdout: '18: tapbd847180-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN \n link/ether fa:16:3e:34:86:5d brd ff:ff:ff:ff:ff:ff\n inet 172.16.59.2/24 brd 172.16.59.255 scope global tapbd847180-7f\n'
Stderr: ''
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.dhcp] Unable to access /var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/pid
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf cat /proc/None/cmdline
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'cat', '/proc/None/cmdline']
Exit code: 99
Stdout: 'Unauthorized command: cat /proc/None/cmdline\n'
Stderr: ''
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay QUANTUM_NETWORK_ID=96e41ff1-abd3-4330-83c6-f5294311cc1f dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapbd847180-7f --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/host --dhcp-optsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,172.16.59.0,static,120s
2012-09-24 11:05:56 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=96e41ff1-abd3-4330-83c6-f5294311cc1f', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tapbd847180-7f', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=set:tag0,172.16.59.0,static,120s']
Exit code: 255
Stdout: ''
Stderr: 'Option "-no-hosts" is unknown, try "ip -help".\n'
2012-09-24 11:05:56 ERROR [quantum.agent.dhcp_agent] Unable to enable dhcp.
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/dhcp_agent.py", line 91, in call_driver
    getattr(driver, action)()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/dhcp.py", line 112, in enable
    self.spawn_process()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/dhcp.py", line 258, in spawn_process
    utils.execute(cmd, self.root_helper)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'QUANTUM_RELAY_SOCKET_PATH=/var/lib/quantum/dhcp/lease_relay', 'QUANTUM_NETWORK_ID=96e41ff1-abd3-4330-83c6-f5294311cc1f', 'dnsmasq', '--no-hosts', '--no-resolv', '--strict-order', '--bind-interfaces', '--interface=tapbd847180-7f', '--except-interface=lo', '--domain=openstacklocal', '--pid-file=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/pid', '--dhcp-hostsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/host', '--dhcp-optsfile=/var/lib/quantum/dhcp/96e41ff1-abd3-4330-83c6-f5294311cc1f/opts', '--dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update', '--leasefile-ro', '--dhcp-range=set:tag0,172.16.59.0,static,120s']
Exit code: 255
Stdout: ''
Stderr: 'Option "-no-hosts" is unknown, try "ip -help".\n'
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] Making asynchronous call on q-plugin ...
2012-09-24 11:05:56 DEBUG [quantum.openstack.common.rpc.amqp] MSG_ID is fb80f9f349554914a1b58939c01c1cfb
2012-09-24 11:05:56 DEBUG [amqplib] Closed channel #1
2012-09-24 11:05:56 DEBUG [amqplib] using channel_id: 1
2012-09-24 11:05:56 DEBUG [amqplib] Channel open

[Endre Karlson (endre-karlson)]

[DEFAULT]
# Show debugging output in log (sets DEBUG log level output)
debug = true

# Where to store dnsmasq state files. This directory must be writable by the
# user executing the agent. The value below is compatible with a default
# devstack installation.
state_path = /var/lib/quantum

# The DHCP agent will resync its state with Quantum to recover from any
# transient notification or rpc errors. The interval is number of
# seconds between attempts.
# resync_interval = 30

# The DHCP requires that an inteface driver be set. Choose the one that best
# matches you plugin.

# OVS
interface_driver = quantum.agent.linux.interface.OVSInterfaceDriver
# LinuxBridge
#interface_driver = quantum.agent.linux.interface.BridgeInterfaceDriver
# Ryu
#interface_driver = quantum.agent.linux.interface.RyuInterfaceDriver

# The agent can use other DHCP drivers. Dnsmasq is the simplest and requires
# no additional setup of the DHCP server.
dhcp_driver = quantum.agent.linux.dhcp.Dnsmasq

# Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y and
# iproute2 package that supports namespaces).
use_namespaces = False

# Use "sudo quantum-rootwrap /etc/quantum/rootwrap.conf" to use the real
# root filter facility.
# Change to "sudo" to skip the filtering and just run the comand directly
root_helper = sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf

[Endre Karlson (endre-karlson)]

If I enable namespaces it works fine though, I only have 1 network and 1 subnet in the tenant network.

[dan wendlandt (danwent)]

according to the man page at least, this option seems to be available in dnsmasq 2.59, which is what I see on ubuntu precise.

probably a good idea to include what version of dnsmasq your system has (as well as your general distro).

[Endre Karlson (endre-karlson)]

Seems to work ok with root_helper = sudo

dpkg -l | grep dnsm
ii dnsmasq 2.59-4 Small caching DNS proxy and DHCP/TFTP server
ii dnsmasq-base 2.59-4 Small caching DNS proxy and DHCP/TFTP server
ii dnsmasq-utils 2.59-4 Utilities for manipulating DHCP leases

Ubuntu 12.04 is my distro.

[Salvatore Orlando (salvatore-orlando)]

It seems the problem lies in the quantum-rootwrap, as the wrong filter is apparently being picked.
From the log it seems 'ip' is being executed instead of 'dnsmasq'.

I am tentatively assigning to Mark, as he's the goto guy for dhcp agent stuff, although this might be a pure rootwrap bug.

Endre confirms that with root_helper = sudo the problem disappears.

[Gary Kotton (garyk)]

I have confirmed.

A simple way to reproduce the problem is as follows:

Works:

 sudo /opt/stack/quantum/bin/quantum-rootwrap /etc/quantum/rootwrap.conf QUANTUM_RELAY_SOCKET_PATH=/opt/stack/data/dhcp/lease_relay QUANTUM_NETWORK_ID=133c6ebb-43ca-4617-a288-466e0353f08f ip netns exec qdhcp-133c6ebb-43ca-4617-a288-466e0353f08f dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap952501ed-cc --except-interface=lo --domain=openstacklocal --pid-file=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/pid --dhcp-hostsfile=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/host --dhcp-optsfile=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/opts --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s

Fails:
sudo /opt/stack/quantum/bin/quantum-rootwrap /etc/quantum/rootwrap.conf QUANTUM_RELAY_SOCKET_PATH=/opt/stack/data/dhcp/lease_relay QUANTUM_NETWORK_ID=133c6ebb-43ca-4617-a288-466e0353f08f dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap952501ed-cc --except-interface=lo --domain=openstacklocal --pid-file=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/pid --dhcp-hostsfile=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/host --dhcp-optsfile=/opt/stack/data/dhcp/133c6ebb-43ca-4617-a288-466e0353f08f/opts --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s

Problem is that the rootwrap filters are defined as follows:

ip_exec_dnsmasq: DnsmasqFilter, /sbin/ip, root
dnsmasq: DnsmasqFilter, /sbin/dnsmasq, root
dnsmasq_usr: DnsmasqFilter, /usr/sbin/dnsmasq, root

So when no namespaces are used the first match is returned.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/13572

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/13572
Committed: http://github.com/openstack/quantum/commit/6194ab2becbd255e6b795c828844db2f3ab85659
Submitter: Jenkins
Branch: master

commit 6194ab2becbd255e6b795c828844db2f3ab85659
Author: Gary Kotton <email address hidden>
Date: Mon Sep 24 12:31:27 2012 +0000

    Fix rootwrap filter for dnsmasq when no namespace is used

    Fixes bug 1055384

    Change-Id: I98381299f28da0e4c443efd4c22ba551022e0288

[yong sheng gong (gongysh)]

Hi Dan, Garyk,
I think the root cause is:
ip_exec_dnsmasq: DnsmasqFilter, /sbin/ip, root
dnsmasq: DnsmasqFilter, /sbin/dnsmasq, root
dnsmasq_usr: DnsmasqFilter, /usr/sbin/dnsmasq, root

most of our filters do not match the needed command with the command defined in rootwrapper filter files.

For this bug, because DnsmasqFilter does not try to match the wanted dnsmasq with the /sbin/ip defined in dhcp.filters file, we select the first rule.

It seems we will allow the user to specify any command to run in rootwrapper filter files. For example, I can define ip_exec_dnsmasq: DnsmasqFilter, /path/to/anyprogramtorun, root although our quantum is trying to run dnsmasq, but we will match and change to run anyprogramtorun.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/13655

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/13655
Committed: http://github.com/openstack/quantum/commit/362bc4d2f49a4e484d27fbee79feebb2d7f03a39
Submitter: Jenkins
Branch: milestone-proposed

commit 362bc4d2f49a4e484d27fbee79feebb2d7f03a39
Author: Gary Kotton <email address hidden>
Date: Mon Sep 24 12:31:27 2012 +0000

    Fix rootwrap filter for dnsmasq when no namespace is used

    Fixes bug 1055384

    Change-Id: I98381299f28da0e4c443efd4c22ba551022e0288