using neutron CLI to set a bad gateway result in dnsmasq continuously restart

Bug #1341042 reported by Pauline Yeung
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Invalid
Medium
ugvddm

Bug Description

I'm using devstack stable/icehouse, and my neutron version is 1409da70959496375f1ccccac45457663a918ec8

I created an internal network not connected to the router using default gateway and dhcp enabled.
I then created a VM which activated dnsmasq, which offered an IP to this VM.

If I mis-update the gateway, by going to Horizon, "Networks", "test-net", "subnet-2" "Edit Subnet", and set
  Gateway IP (optional): 10.100.100.100
or mis-update the gateway using neutron CLI,
it results in neutron-dhcp-agent continously restarting the dnsmasq for the "private" network.

Horizon will not allow the user to mis-update the gateway, if I set "force_gateway_on_subnet = True" in /etc/neutron/neutron.conf.
But this parameter does not affect the behavior of the neutron CLI.

> neutron net-create test-net
Created a new network:
+----------------+--------------------------------------+
| Field | Value |
+----------------+--------------------------------------+
| admin_state_up | True |
| id | b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a |
| name | test-net |
| shared | False |
| status | ACTIVE |
| subnets | |
| tenant_id | 8092813be8fd4122a20ee3a6bfe91162 |
+----------------+--------------------------------------+

> neutron net-list
+--------------------------------------+----------+--------------------------------------------------+
| id | name | subnets |
+--------------------------------------+----------+--------------------------------------------------+
| 0dd5722d-f535-42ec-9257-437c05e4de25 | private | 81859ee5-4ea5-4e60-ab2a-ba74146d39ba 10.0.0.0/24 |
| 27c1649d-f6fc-4893-837d-dbc293fc4b80 | public | 6c1836a1-eb7d-4acb-ad6f-6c394cedced5 |
| b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a | test-net | |
+--------------------------------------+----------+--------------------------------------------------+

> neutron subnet-create --name subnet-2 test-net --enable_dhcp=True 10.10.150.0/24
Created a new subnet:
+------------------+--------------------------------------------------+
| Field | Value |
+------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.10.150.2", "end": "10.10.150.254"} |
| cidr | 10.10.150.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 10.10.150.1 |
| host_routes | |
| id | 4e462af6-7514-413c-bf55-4ae9a1643ce9 |
| ip_version | 4 |
| name | subnet-2 |
| network_id | b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a |
| tenant_id | 8092813be8fd4122a20ee3a6bfe91162 |
+------------------+--------------------------------------------------+

> neutron subnet-list
+--------------------------------------+----------------+----------------+--------------------------------------------------+
| id | name | cidr | allocation_pools |
+--------------------------------------+----------------+----------------+--------------------------------------------------+
| 4e462af6-7514-413c-bf55-4ae9a1643ce9 | subnet-2 | 10.10.150.0/24 | {"start": "10.10.150.2", "end": "10.10.150.254"} |
| 81859ee5-4ea5-4e60-ab2a-ba74146d39ba | private-subnet | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
+--------------------------------------+----------------+----------------+--------------------------------------------------+

> ps -ef |grep dnsmasq | grep -v grep
nobody 995 635 0 16:09 ? 00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
nobody 12991 1 0 18:21 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap75786806-5a --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/host --addn-hosts=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

> nova boot --image cirros-0.3.1-x86_64-uec --flavor m1.tiny --nic net-id=b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a vm2
:
:

> nova list
+--------------------------------------+------+--------+------------+-------------+----------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------------------+
| d98511f7-452c-4ab6-8af9-d73576714c87 | vm1 | ACTIVE | - | Running | private=10.0.0.2 |
| 5934a236-edfc-4662-85f4-f00bdac1137a | vm2 | ACTIVE | - | Running | test-net=10.10.150.2 |
+--------------------------------------+------+--------+------------+-------------+----------------------+

> ps -ef | grep dnsmasq | grep -v grep
nobody 995 635 0 16:09 ? 00:00:00 /usr/sbin/dnsmasq --no-resolv --keep-in-foreground --no-hosts --bind-interfaces --pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid --listen-address=127.0.1.1 --conf-file=/var/run/NetworkManager/dnsmasq.conf --cache-size=0 --proxy-dnssec --enable-dbus=org.freedesktop.NetworkManager.dnsmasq --conf-dir=/etc/NetworkManager/dnsmasq.d
nobody 12991 1 0 18:21 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap75786806-5a --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/host --addn-hosts=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/0dd5722d-f535-42ec-9257-437c05e4de25/opts --leasefile-ro --dhcp-range=tag0,10.0.0.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal
nobody 25284 1 0 19:02 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap41fe641d-7c --except-interface=lo --pid-file=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/pid --dhcp-hostsfile=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/host --addn-hosts=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/addn_hosts --dhcp-optsfile=/opt/stack/data/neutron/dhcp/b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a/opts --leasefile-ro --dhcp-range=tag0,10.10.150.0,static,86400s --dhcp-lease-max=256 --conf-file= --domain=openstacklocal

19:17:42 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
995
12991
25284

> neutron subnet-update 4e462af6-7514-413c-bf55-4ae9a1643ce9 --gateway_ip=10.100.100.100
Updated subnet: 4e462af6-7514-413c-bf55-4ae9a1643ce9

19:19:36 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
995
25284
30892

19:19:49 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
995
25284
30999

19:19:59 > ps -ef | grep dnsmasq | grep -v grep | awk '{print $2}'
995
25284
31137

I get the following output from screen:

2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/dhcp_agent.py", line 129, in call_driver
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent getattr(driver, action)(**action_kwargs)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 165, in enable
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent reuse_existing=True)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 848, in setup
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent self._set_default_route(network, interface_name)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/dhcp.py", line 731, in _set_default_route
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent device.route.add_gateway(subnet.gateway_ip)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 369, in add_gateway
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent self._as_root(*args)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 218, in _as_root
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent kwargs.get('use_root_namespace', False))
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 71, in _as_root
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent namespace)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/ip_lib.py", line 82, in _execute
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent root_helper=root_helper)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent File "/opt/stack/neutron/neutron/agent/linux/utils.py", line 76, in execute
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent raise RuntimeError(m)
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Command: ['sudo', '/usr/local/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-b7bb10bb-48e0-4c1a-a5fc-9590b6619f5a', 'ip', 'route', 'replace', 'default', 'via', '10.100.100.100', 'dev', 'tap41fe641d-7c']
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Network is unreachable\n'
2014-07-11 19:23:26.212 TRACE neutron.agent.dhcp_agent

ugvddm (271025598-9)
Changed in neutron:
assignee: nobody → ugvddm (271025598-9)
Changed in neutron:
importance: Undecided → Medium
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

Apparently the failure to setup a default gateway leads to a resync for a given subnet, which in turn leads to dnsmasq respawn

ugvddm (271025598-9)
Changed in neutron:
status: New → Confirmed
Revision history for this message
ugvddm (271025598-9) wrote :
Changed in neutron:
status: Confirmed → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.