router:dhcp ports are open resolvers

Bug #1501206 reported by Tore Anderson on 2015-09-30
40
This bug affects 6 people
Affects Status Importance Assigned to Milestone
OpenStack Security Advisory
Undecided
Unassigned
neutron
High
David Homolka
neutron (Ubuntu)
High
new

Bug Description

When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/:

«Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s.

[...]

What can I do?

If you operate a DNS server, please check the settings.

Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch»

It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others.

Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too.

For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so.

Tore

Jeremy Stanley (fungi) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Changed in ossa:
status: New → Incomplete
description: updated

While this is surely a bug, I wonder how the DNS responders operating in the network namespace were reachable from the Internet.

Living in a private network they're supposed to be isolated. If this does not happen it could be because:

- a deployment issue, and therefore it is likely a documentation bug should be filed as well
- a technical defect, in which case this should be granted an OSSA immediately and we'll probably need to look into other issues where services running in network namespaces can be reached from the Internet.

Jeremy Stanley (fungi) wrote :

Based on the bug description, Neutron was configured to provide DHCP (and so also DNS) within a globally-routable network without a separate packet filter between it and the Internet at large. The reporter's expectation seems to be that OpenStack should provide some filtering/client access restriction by default to mitigate this.

Tore Anderson (toreanderson) wrote :

I can confirm Jeremy's understanding. The tenant in question has opted not to use private (RFC1918) IPv4 addressing or SNAT. Instead, he has a range of public IPv4 addresses which us routed from the upstream network to his router's address in the external gateway network. This allows the tenant to configure his subnets using prefixes from this public range. That said, the use case for enabling DHCP service (and by extension DNS service) on his subnets remains the same as for any other tenant.

I would also like to note that for IPv6, the method described above is the only possible way do it. There is no SNAT for IPv6; using private IPv6 address space on a subnet will thus only ensure that instances on that subnet cannot communicate with the world outside of OpenStack *at all*.

So I would agree with Salvatore's last comment - you cannot assume that the external network will ensure that services running in network namespaces (such as dnsmasq) is not reachable from the global Internet. Those services, or their containing namespaces, must themselves ensure that they do not inadvertently offer their services to external non-OpenStack clients.

Finally, I can confirm that there is no firewall or packet filter in the network outside of OpenStack. The expectation is that the tenants will maintain their own firewall rules using security groups (or maybe FWaaS) - but these do not apply to the qdchp network namespace. Also, it is not really a workable solution for an operator to simply drop all inbound DNS traffic (dst port 53/udp) before it reaches OpenStack, as this would also interfere with other completely legitimate use cases (e.g., another tenant running a non-recursing authoritative DNS server on one of his instances).

Mark McClain (markmcclain) wrote :

In the scenario, outlined above using DHCP on a publicly routable network is a known limitation of the current setup. dnsmasq is not a recursive resolver (it's a forwarder). We should tighten the rules and I would be happy to work on it unless someone has a patch already.

The question is whether the bug should be embargoed. I'd lean towards working on this in the open since is an easily discoverable situation and known limitation of using the DHCP service on a public segment. I don't believe that haring this information with others would lead to increased DNS attacks on existing deployments since scanning tools already exist to find vulnerable DNS servers. Once a fix is available in stable branches we should notify interested parties.

Tore Anderson (toreanderson) wrote :

I just received notification that bug #1498665 has been fixed. While I haven't yet confirmed that the fix is good myself, assuming that it is means that this bug now also applies generally to most IPv6 subnet with DHCP service (since IPv6 is generally be deployed using public addresses).

For the record I have no strong opinion about keeping the bug embargoed or not. I do agree with Mark that making it public is not likely to significantly increase the risk to OpenStack operators though, as folks looking for open resolvers will likely just scan the entire IPv4 address space rather than bothering to specifically target networks that are known to run OpenStack (or other vulnerable-by-default products).

If there are no objections, let's open this bug by the end of the week.

Jeremy Stanley (fungi) wrote :

It's worth noting that this thread http://comments.gmane.org/gmane.network.dns.dnsmasq.general/7778 from a couple years ago suggests that there's no mechanism to limit dnsmasq's resolver to specific client addresses/ranges either (unless that's been implemented more recently?), so chances are this will need to be solved with a packet filter somewhere.

Changed in neutron:
importance: Undecided → High
assignee: nobody → Mark McClain (markmcclain)
Mark McClain (markmcclain) wrote :

@fungi: Right my was plan was to use the current iptables machinery to restrict inbound DNS traffic to known subnets for a network.

Switching this bug report to public.

description: updated
information type: Private Security → Public Security
Kevin Benton (kevinbenton) wrote :

I'm not particularly a fan of even classifying this as a public security bug. It's not a vulnerability in Neutron in that it doesn't allow you to exploit Neutron or OpenStack in any particular way.

Amplification attacks are a result of ISPs failing to filter spoofed IP traffic. The open resolver project has taken to launching a crusade against people offering DNS lookup services exposed to the Internet; however, it won't really solve the issue in the long run because the amplification attacks can be adjusted to just leverage authoritative DNS nameservers or any other myriad of UDP-based services.

I'm for fixing this, but I don't think it should be handled under the priority of a vulnerability... /rant

Miguel Angel Ajo (mangelajo) wrote :

I agree with kevin on this.

Alright, removing the security class and closing the OSSA task.

Changed in ossa:
status: Incomplete → Won't Fix
information type: Public Security → Public
Benoît Knecht (benoit-knecht) wrote :

I have submitted a fix for review: https://review.openstack.org/#/c/256280/.

Changed in neutron:
assignee: Mark McClain (markmcclain) → Cedric Brandily (cbrandily)
status: New → In Progress
Changed in neutron:
assignee: Cedric Brandily (cbrandily) → Benoît Knecht (benoit-knecht)
Changed in neutron:
milestone: none → mitaka-rc1
Changed in neutron:
milestone: mitaka-rc1 → newton-1

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/256280
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: In Progress → Confirmed
assignee: Benoît Knecht (benoit-knecht) → nobody
milestone: newton-1 → none
Tore Anderson (toreanderson) wrote :

Just for the record, this bug is still present in Mitaka.

I'd also like to point out that it occurs on provider networks as well, e.g., if you have a flat/vlan network with a router external from OpenStack (no L3 agent) but let OpenStack run the DHCP service. In that case, the dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire IPv4/IPv6 address space, and will likely be abused to death for DDoS amplification by the various botnets once they discover its existence.

Changed in neutron:
assignee: nobody → Benoît Knecht (benoit-knecht)
status: Confirmed → In Progress
Dr. Jens Harbott (j-harbott) wrote :

IMO instead of setting up some complicated filtering, the proper solution would be to run dnsmasq with the option

--local-service
    Accept DNS queries only from hosts whose address is on a local subnet, ie a subnet for which an interface exists on the server. This option only has effect is there are no --interface --except-interface, --listen-address or --auth-server options. It is intended to be set as a default on installation, to allow unconfigured installations to be useful but also safe from being used for DNS amplification attacks.

(see http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html)

The only issue here is that currently started with a lot of --interfaces and --exclude-interfaces options, that result in the above option being ignored. At least in my enviroment, though, all of these options are redundant and I can replace them with --local-service and get the bug fixed.

Fix proposed to branch: master
Review: https://review.openstack.org/379837

Changed in neutron:
assignee: Benoît Knecht (benoit-knecht) → Dustin Lundquist (dlundquist)

Change abandoned by Dustin Lundquist (<email address hidden>) on branch: master
Review: https://review.openstack.org/379837
Reason: Duplicate

Changed in neutron:
assignee: Dustin Lundquist (dlundquist) → Dr. Jens Rosenboom (j-rosenboom-j)

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/256280
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: In Progress → Incomplete
assignee: Dr. Jens Rosenboom (j-rosenboom-j) → nobody
tags: added: l3-bgp
tags: removed: dns
Sreekumar S (sreesiv) wrote :

Is this not 'In progress' with https://review.openstack.org/#/c/333829/???

Changed in neutron:
status: Incomplete → In Progress
status: In Progress → Incomplete

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/333829
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Changed in neutron:
status: Incomplete → Confirmed
Changed in neutron:
assignee: nobody → David Homolka (davidhomolka)
status: Confirmed → In Progress
James Page (james-page) wrote :

Marking Ubuntu task as Invalid; Ubuntu will pickup whatever ends up being landed into Neutron itself via Queens and other stable point releases.

Changed in neutron (Ubuntu):
status: New → Triaged
importance: Undecided → High
status: Triaged → Invalid
new (cloudie) on 2018-03-02
Changed in neutron (Ubuntu):
assignee: nobody → new (cloudie)
status: Invalid → In Progress
Atif (asaeed) on 2018-03-06
Changed in neutron (Ubuntu):
status: In Progress → Confirmed
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers