router:dhcp ports are open resolvers

[Tore Anderson]

When configuring an public IPv4 subnet with DHCP enabled inside Neutron (and attaching it to an Internet-connected router), the DNS recursive resolver service provided by dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire Internet. This is a huge problem from a security standpoint, as open resolvers are very likely to be abused for DDoS purposes. This does not only cause significant damage to third parties (i.e., the true destination of the DDoS attack and every network in between), but also on the local network or servers (due to saturation of all the available network bandwidth and/or the processing capacity of the node running the dnsmasq instance). Quoting from http://openresolverproject.org/:

«Open Resolvers pose a significant threat to the global network infrastructure by answering recursive queries for hosts outside of its domain. They are utilized in DNS Amplification attacks and pose a similar threat as those from Smurf attacks commonly seen in the late 1990s.

[...]

What can I do?

If you operate a DNS server, please check the settings.

Recursive servers should be restricted to your enterprise or customer IP ranges to prevent abuse. Directions on securing BIND and Microsoft nameservers can be found on the Team CYMRU Website - If you operate BIND, you can deploy the TCP-ANY patch»

It seems reasonable to expect that the dnsmasq instance within Neutron would only respond to DNS queries from the subnet prefixes it is associated with and ignore all others.

Note that this only occurs for IPv4. That is however likely just a symptom of bug #1499170, which breaks all IPv6 DNS queries (external as well as internal). I would assume that when bug #1499170 is fixed, the router:dhcp ports will immediately start being open resolvers over IPv6 too.

For what it's worth, the reason I noticed this issue in the first place was that NorCERT (the national Norwegian Computer Emergency Response Team - http://www.cert.no/) got in touch with us, notifying us about the open resolvers they had observed in our network and insisted that we lock them down ASAP. It only took NorCERT couple of days after the subnet was first created to do so.

Tore

[Jeremy Stanley (fungi)]

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

[Salvatore Orlando (salvatore-orlando)]

While this is surely a bug, I wonder how the DNS responders operating in the network namespace were reachable from the Internet.

Living in a private network they're supposed to be isolated. If this does not happen it could be because:

- a deployment issue, and therefore it is likely a documentation bug should be filed as well
- a technical defect, in which case this should be granted an OSSA immediately and we'll probably need to look into other issues where services running in network namespaces can be reached from the Internet.

[Jeremy Stanley (fungi)]

Based on the bug description, Neutron was configured to provide DHCP (and so also DNS) within a globally-routable network without a separate packet filter between it and the Internet at large. The reporter's expectation seems to be that OpenStack should provide some filtering/client access restriction by default to mitigate this.

[Tore Anderson (toreanderson)]

I can confirm Jeremy's understanding. The tenant in question has opted not to use private (RFC1918) IPv4 addressing or SNAT. Instead, he has a range of public IPv4 addresses which us routed from the upstream network to his router's address in the external gateway network. This allows the tenant to configure his subnets using prefixes from this public range. That said, the use case for enabling DHCP service (and by extension DNS service) on his subnets remains the same as for any other tenant.

I would also like to note that for IPv6, the method described above is the only possible way do it. There is no SNAT for IPv6; using private IPv6 address space on a subnet will thus only ensure that instances on that subnet cannot communicate with the world outside of OpenStack *at all*.

So I would agree with Salvatore's last comment - you cannot assume that the external network will ensure that services running in network namespaces (such as dnsmasq) is not reachable from the global Internet. Those services, or their containing namespaces, must themselves ensure that they do not inadvertently offer their services to external non-OpenStack clients.

Finally, I can confirm that there is no firewall or packet filter in the network outside of OpenStack. The expectation is that the tenants will maintain their own firewall rules using security groups (or maybe FWaaS) - but these do not apply to the qdchp network namespace. Also, it is not really a workable solution for an operator to simply drop all inbound DNS traffic (dst port 53/udp) before it reaches OpenStack, as this would also interfere with other completely legitimate use cases (e.g., another tenant running a non-recursing authoritative DNS server on one of his instances).

[Mark McClain (markmcclain)]

In the scenario, outlined above using DHCP on a publicly routable network is a known limitation of the current setup. dnsmasq is not a recursive resolver (it's a forwarder). We should tighten the rules and I would be happy to work on it unless someone has a patch already.

The question is whether the bug should be embargoed. I'd lean towards working on this in the open since is an easily discoverable situation and known limitation of using the DHCP service on a public segment. I don't believe that haring this information with others would lead to increased DNS attacks on existing deployments since scanning tools already exist to find vulnerable DNS servers. Once a fix is available in stable branches we should notify interested parties.

[Tore Anderson (toreanderson)]

I just received notification that bug #1498665 has been fixed. While I haven't yet confirmed that the fix is good myself, assuming that it is means that this bug now also applies generally to most IPv6 subnet with DHCP service (since IPv6 is generally be deployed using public addresses).

For the record I have no strong opinion about keeping the bug embargoed or not. I do agree with Mark that making it public is not likely to significantly increase the risk to OpenStack operators though, as folks looking for open resolvers will likely just scan the entire IPv4 address space rather than bothering to specifically target networks that are known to run OpenStack (or other vulnerable-by-default products).

[Tristan Cacqueray (tristan-cacqueray)]

If there are no objections, let's open this bug by the end of the week.

[Jeremy Stanley (fungi)]

It's worth noting that this thread http://comments.gmane.org/gmane.network.dns.dnsmasq.general/7778 from a couple years ago suggests that there's no mechanism to limit dnsmasq's resolver to specific client addresses/ranges either (unless that's been implemented more recently?), so chances are this will need to be solved with a packet filter somewhere.

[Mark McClain (markmcclain)]

@fungi: Right my was plan was to use the current iptables machinery to restrict inbound DNS traffic to known subnets for a network.

[Tristan Cacqueray (tristan-cacqueray)]

Switching this bug report to public.

[Kevin Benton (kevinbenton)]

I'm not particularly a fan of even classifying this as a public security bug. It's not a vulnerability in Neutron in that it doesn't allow you to exploit Neutron or OpenStack in any particular way.

Amplification attacks are a result of ISPs failing to filter spoofed IP traffic. The open resolver project has taken to launching a crusade against people offering DNS lookup services exposed to the Internet; however, it won't really solve the issue in the long run because the amplification attacks can be adjusted to just leverage authoritative DNS nameservers or any other myriad of UDP-based services.

I'm for fixing this, but I don't think it should be handled under the priority of a vulnerability... /rant

[Miguel Angel Ajo (mangelajo)]

I agree with kevin on this.

[Tristan Cacqueray (tristan-cacqueray)]

Alright, removing the security class and closing the OSSA task.

[Benoît Knecht (benoit-knecht)]

I have submitted a fix for review: https://review.openstack.org/#/c/256280/.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/256280
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Tore Anderson (toreanderson)]

Just for the record, this bug is still present in Mitaka.

I'd also like to point out that it occurs on provider networks as well, e.g., if you have a flat/vlan network with a router external from OpenStack (no L3 agent) but let OpenStack run the DHCP service. In that case, the dnsmasq inside the qdhcp network namespace will respond to DNS queries from the entire IPv4/IPv6 address space, and will likely be abused to death for DDoS amplification by the various botnets once they discover its existence.

[Dr. Jens Harbott (j-harbott)]

IMO instead of setting up some complicated filtering, the proper solution would be to run dnsmasq with the option

--local-service
    Accept DNS queries only from hosts whose address is on a local subnet, ie a subnet for which an interface exists on the server. This option only has effect is there are no --interface --except-interface, --listen-address or --auth-server options. It is intended to be set as a default on installation, to allow unconfigured installations to be useful but also safe from being used for DNS amplification attacks.

(see http://www.thekelleys.org.uk/dnsmasq/docs/dnsmasq-man.html)

The only issue here is that currently started with a lot of --interfaces and --exclude-interfaces options, that result in the above option being ignored. At least in my enviroment, though, all of these options are redundant and I can replace them with --local-service and get the bug fixed.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/379837

[OpenStack Infra (hudson-openstack)]

Change abandoned by Dustin Lundquist (<email address hidden>) on branch: master
Review: https://review.openstack.org/379837
Reason: Duplicate

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/256280
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[Sreekumar S (sreesiv)]

Is this not 'In progress' with https://review.openstack.org/#/c/333829/???

[OpenStack Infra (hudson-openstack)]

Change abandoned by Armando Migliaccio (<email address hidden>) on branch: master
Review: https://review.openstack.org/333829
Reason: This review is > 4 weeks without comment, and failed Jenkins the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

[James Page (james-page)]

Marking Ubuntu task as Invalid; Ubuntu will pickup whatever ends up being landed into Neutron itself via Queens and other stable point releases.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333829
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=0fce3ca2c1641fbcfb8327a86d7225e2c3972263
Submitter: Zuul
Branch: master

commit 0fce3ca2c1641fbcfb8327a86d7225e2c3972263
Author: Jens Harbott <email address hidden>
Date: Mon Oct 29 17:08:33 2018 +0000

    Secure dnsmasq process against external abuse

    Currently any dhcp agent instance will work as an open resolver. For
    deployments using publicly routed addresses for tenant networks, this
    allows the agent being abused in dDoS attacks, see [1].

    By setting the `--local-service` option dnsmasq will filter DNS queries
    and reply only to queries from directly attached networks.

    [1] https://bugs.launchpad.net/neutron/+bug/1501206

    Closes-Bug: 1501206
    Change-Id: I76d810aad2ce0f15a88bd798963012fa0efca74e

[Dmitriy Rabotyagov (noonedeadpunk)]

Probably this bugfix is worth backporting at least to rocky?

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 14.0.0.0b1 development milestone.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/633207

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/633210

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/633211

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633210
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=f599c15e33f72d44a18f10cd71a0fc9b13b35080
Submitter: Zuul
Branch: stable/queens

commit f599c15e33f72d44a18f10cd71a0fc9b13b35080
Author: Jens Harbott <email address hidden>
Date: Mon Oct 29 17:08:33 2018 +0000

    Secure dnsmasq process against external abuse

    Currently any dhcp agent instance will work as an open resolver. For
    deployments using publicly routed addresses for tenant networks, this
    allows the agent being abused in dDoS attacks, see [1].

    By setting the `--local-service` option dnsmasq will filter DNS queries
    and reply only to queries from directly attached networks.

    [1] https://bugs.launchpad.net/neutron/+bug/1501206

    Conflicts:
        neutron/cmd/sanity_check.py

    Closes-Bug: 1501206
    Change-Id: I76d810aad2ce0f15a88bd798963012fa0efca74e
    (cherry picked from commit 0fce3ca2c1641fbcfb8327a86d7225e2c3972263)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633211
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=72d9c3ccb34f5c5abb8de0b32d4ef1660b9f502f
Submitter: Zuul
Branch: stable/pike

commit 72d9c3ccb34f5c5abb8de0b32d4ef1660b9f502f
Author: Jens Harbott <email address hidden>
Date: Mon Oct 29 17:08:33 2018 +0000

    Secure dnsmasq process against external abuse

    Currently any dhcp agent instance will work as an open resolver. For
    deployments using publicly routed addresses for tenant networks, this
    allows the agent being abused in dDoS attacks, see [1].

    By setting the `--local-service` option dnsmasq will filter DNS queries
    and reply only to queries from directly attached networks.

    [1] https://bugs.launchpad.net/neutron/+bug/1501206

    Conflicts:
        neutron/cmd/sanity_check.py

    Closes-Bug: 1501206
    Change-Id: I76d810aad2ce0f15a88bd798963012fa0efca74e
    (cherry picked from commit 0fce3ca2c1641fbcfb8327a86d7225e2c3972263)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/633207
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a7afd6e86d833ea44fc17528158e6819618d07f7
Submitter: Zuul
Branch: stable/rocky

commit a7afd6e86d833ea44fc17528158e6819618d07f7
Author: Jens Harbott <email address hidden>
Date: Mon Oct 29 17:08:33 2018 +0000

    Secure dnsmasq process against external abuse

    Currently any dhcp agent instance will work as an open resolver. For
    deployments using publicly routed addresses for tenant networks, this
    allows the agent being abused in dDoS attacks, see [1].

    By setting the `--local-service` option dnsmasq will filter DNS queries
    and reply only to queries from directly attached networks.

    [1] https://bugs.launchpad.net/neutron/+bug/1501206

    Conflicts:
        neutron/cmd/sanity_check.py

    Closes-Bug: 1501206
    Change-Id: I76d810aad2ce0f15a88bd798963012fa0efca74e
    (cherry picked from commit 0fce3ca2c1641fbcfb8327a86d7225e2c3972263)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Brian Haley (<email address hidden>) on branch: master
Review: https://review.openstack.org/523343
Reason: https://review.openstack.org/#/c/333829/ merged

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 11.0.7 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 13.0.3 release.

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 12.0.6 release.