wrong gateway_ip setting in subnet constantly resyncs dhcp-agent, causing DHCP NAK dropping connection in guests

[Miguel Angel Ajo]

If any tenant creates a subnet with a wrong gateway ip (outside the subnet CIDR),
ip route will fail to set the route into the qdhcp namespace, and that
will cause constant resyncs and neutron-dhcp-agent malfunction.

This needs a fix either in parameter acceptance in API, or/and before calling iproute
into the neutron-dhcp-agent to avoid this problem.

[root@controllerNN ~(keystone_admin)]# neutron subnet-create privnet2 192.168.126.0/24 --gateway 192.168.222.1 --name badsubnet
Created a new subnet:
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| allocation_pools | {"start": "192.168.126.1", "end": "192.168.126.254"} |
| cidr | 192.168.126.0/24 |
| dns_nameservers | |
| enable_dhcp | True |
| gateway_ip | 192.168.222.1 |
| host_routes | |
| id | 4f41041f-3035-43f1-9dbf-daa61a96fb7e |
| ip_version | 4 |
| name | badsubnet |
| network_id | 9ef49c17-6845-4e26-827c-bb2423317f67 |
| tenant_id | 60b2f5dcf34542f7952216beb97793a1 |
+------------------+------------------------------------------------------+

And this is the log of the dhcp server:

[root@controllerNN ~(keystone_admin)]# tail -f /var/log/neutron/dhcp-agent.log
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent root_helper=root_helper)
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent raise RuntimeError(m)
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-9ef49c17-6845-4e26-827c-bb2423317f67', 'ip', 'route', 'replace', 'default', 'via', '192.168.222.1', 'dev', 'tap8ac47b9f-ea']
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Network is unreachable\n'
2014-06-22 08:41:08.107 3767 TRACE neutron.agent.dhcp_agent
2014-06-22 08:41:08.108 3767 INFO neutron.agent.dhcp_agent [-] Synchronizing state complete
2014-06-22 08:41:38.109 3767 INFO neutron.agent.dhcp_agent [-] Synchronizing state
2014-06-22 08:41:38.426 3767 ERROR neutron.agent.dhcp_agent [-] Unable to enable dhcp for 9ef49c17-6845-4e26-827c-bb2423317f67.
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent Traceback (most recent call last):
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/dhcp_agent.py", line 127, in call_driver
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent getattr(driver, action)(**action_kwargs)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 166, in enable
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent reuse_existing=True)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 854, in setup
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent self._set_default_route(network, port)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/dhcp.py", line 737, in _set_default_route
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent device.route.add_gateway(subnet.gateway_ip)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 368, in add_gateway
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent self._as_root(*args)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 217, in _as_root
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent kwargs.get('use_root_namespace', False))
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 70, in _as_root
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent namespace)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_lib.py", line 81, in _execute
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent root_helper=root_helper)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent File "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py", line 76, in execute
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent raise RuntimeError(m)
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent RuntimeError:
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qdhcp-9ef49c17-6845-4e26-827c-bb2423317f67', 'ip', 'route', 'replace', 'default', 'via', '192.168.222.1', 'dev', 'tap8ac47b9f-ea']
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent Exit code: 2
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent Stdout: ''
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent Stderr: 'RTNETLINK answers: Network is unreachable\n'
2014-06-22 08:41:38.426 3767 TRACE neutron.agent.dhcp_agent
2014-06-22 08:41:38.427 3767 INFO neutron.agent.dhcp_agent [-] Synchronizing state complete

[Miguel Angel Ajo (mangelajo)]

I think this bug report duplicates this:

https://review.openstack.org/#/c/92619/

https://bugs.launchpad.net/neutron/+bug/1304181

just it's more severe than it looks.

[Jeremy Stanley (fungi)]

I've set the security advisory task incomplete pending additional details on the scope of affected releases and potential risk/impact.

The linked review suggests this behavior was introduced with c3706fa2 between grizzly milestone 1 and 2 and also backported to folsom between the 2012.2.2 and 2012.2.3 stable point releases.

The risk appears to be that one tenant could launch a volume-based denial of service against all tenants sharing a common DHCP service.

The other possible duplicate bug doesn't mention its potential security impact, so there may still be reason to keep this report embargoed while a fix is implemented.

The proposed fix linked above has been in a work-in-progress state since before the Juno summit, and so will probably need to be revisited to get it into shape. We will also need backports to havana and icehouse, but the patch looks relatively trivial so hopefully this doesn't pose much of a challenge.

[Thierry Carrez (ttx)]

looks valid. Could neutron-coresec have a look and confirm the security impact ?

[Aaron Rosen (arosen)]

Hi,

To me it looks like this doesn't cause the dhcp-agent to constantly resync in the current codebase:

https://github.com/openstack/neutron/blob/master/neutron/agent/dhcp_agent.py#L171 <--- in sync which calls this method

Then:

https://github.com/openstack/neutron/blob/master/neutron/agent/dhcp_agent.py#L216

^ this catches RuntimeError which is raised so it seems like it won't keep resyncing and the agent will continue? Am I missing something here?

[Tristan Cacqueray (tristan-cacqueray)]

@arosen, thanks for the analysis!
 so the question is, even in case of bad configuration (throwing RuntimeError I guess), will the dhcp service still be functional ? At least for other tenants ?

[Aaron Rosen (arosen)]

It looks like it to me. I'll take this bug though and check.

[Aaron Rosen (arosen)]

Hi, I just looked into this and reproduced this. This causes the code to TRACE and resync but it does not break dhcp from continuing from working. In my opinion I don't think this is worth issuing a Security Advisory

[Thierry Carrez (ttx)]

@Miguel, do you agree with Aaron's findings ? If yes, we should probably open this bug and fix it 'without a security advisory) publicly.

[Thierry Carrez (ttx)]

Note; we'll open this unless someone complains over the next few days.

[Oleg Bondarev (obondarev)]

This was fixed with https://review.openstack.org/#/c/92619