Designate mDNS DoS through incorrect handling of large RecordSets

[Florian Weimer]

Designate does not enforce the DNS protocol limit concerning record set sizes when adding records for a domain name. The protocol limit is slightly less than 2**16 bytes because of some data that must accompany any record set, as part of the zone file transfer.

As a result, the rendering loop in desginate-mdns can does not make progress because the problematic record set at rrsets[i] will never fit (per designate/mdns/handler.py:RequestHandler._handle_axfr):

        # Render the results, yielding a packet after each TooBig exception.
        i, renderer = 0, None
        while i < len(rrsets):
            # No renderer? Build one
            if renderer is None:
                renderer = dns.renderer.Renderer(
                    response.id, response.flags, max_message_size)
…
            try:
                renderer.add_rrset(dns.renderer.ANSWER, rrsets[i])
                i += 1
            except dns.exception.TooBig:
                renderer.write_header()
                if request.had_tsig:
…
                yield renderer
                renderer = None

When this happens, designate-mdns will keep sending SOA records over and over again:

sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31
sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31
sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31

Here is how I triggered this. I created a domain example.org:

+--------------------------------------+--------------+------------+
| id | name | serial |
+--------------------------------------+--------------+------------+
| ded903fc-60ef-4b5e-a37c-f3181baf9516 | example.org. | 1435315435 |
+--------------------------------------+--------------+------------+

Then I added several hundred NS records for the same subdomain, using this shell command:

for x in {1..304} ; do
  designate record-create --name 'sub.example.org.' --type NS --data \
$x'.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.'$x. \
  ded903fc-60ef-4b5e-a37c-f3181baf9516
done

In the tested configuration, PowerDNS will attempt a zone file transfer, and trigger the bug. Because it keeps receiving data, it does not seem it will ever run into a timeout (and if it does, it will try again).

Downstream bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1236014

[Kiall Mac Innes (kiall)]

Attached a patch providing an initial candidate fix, based off master.

Tests are passing, but this has not yet been validated with BIND/PowerDNS etc

[Kiall Mac Innes (kiall)]

Apologies, patch is based off https://review.openstack.org/200230 rather than master

[Kiall Mac Innes (kiall)]

Finally, this fixes the DOS - Not the failure to apply quotas..

[Kiall Mac Innes (kiall)]

Affects debian: https://packages.debian.org/stretch/designate
Affects ubuntu: http://packages.ubuntu.com/vivid/designate

RedHat doesn't seem to have an equivalent package search site, but I'm assuming since they filed this issue, it affects RedHat too.

[Graham Hayes (grahamhayes)]

I think there is 2 parts to this bug:

1: Quotas were being bypassed as part of the v1 API.
2. If there was enough RRs in a RRSet MiniDNS went into a loop.
3. MiniDNS does not have a timeout.

1 is not really relevant - 2 should not happen in anycase.

The patch fixes 2, but not 3 - should we be include that in the fix?

[Kiall Mac Innes (kiall)]

I've gone and implemented the RRSet related quota enforcement for Master and Kilo, this was not previously implemented. I've included them in this bug, as publishing them now might indicate there's an issue with RRset sizes, but I don't consider them directly part of this DoS vulnerability.

i.e. An operator could choose to have infinite quotas (a valid use case), and we should support that.

[Kiall Mac Innes (kiall)]

Attached initial OSSA yaml based on existing documents from https://github.com/openstack/ossa/tree/master/ossa

[Graham Hayes (grahamhayes)]

Updated Kilo patch

[Kiall Mac Innes (kiall)]

All patches have now been validated as fixing the DoS.

[Graham Hayes (grahamhayes)]

Main Patch:

Bind9 + Master - Fix Verified
Bind9 + Kilo - Fix Verified

I have not tested the Quotas patch yet, but it should be DNS Server agnostic.

[Kiall Mac Innes (kiall)]

Attached reproduction script used.

Ran before applying patches, mDNS will consume 100% CPU once it's completed and zones will not propagate to the nameservers.

Ran after applying patches, mDNS will correctly propagate zones to the nameservers.

[Tim Simmons (timsim)]

Verified main fix with:
Bind9 + Master
Bind9 + Kilo

Verified quotas fix on:
Master
Kilo

[Kiall Mac Innes (kiall)]

CVE # Allocation request sent, based on VMT template. Copy of email:

A vulnerability was discovered in OpenStack Designate (see below).
In order to ensure full traceability, we need a CVE number assigned
that we can attach to private and public notifications. Please treat
the following information as confidential until further public
disclosure.

Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: OpenStack Designate
Affects: OpenStack Designate version 2015.1.0

Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max
allowed DNS packet size, an authenticated user may cause the
Designate mDNS service to enter an infinite loop, triggering a DoS.

Thanks in advance,
Kiall Mac Innes

[Kiall Mac Innes (kiall)]

No CVE number yet, went ahead and notified downstreams. Disclosure date set for 2015-07-28, 1500UTC.

Copy of email below:

This is an advance warning of a vulnerability discovered in OpenStack
Designate, to give you, as downstream stakeholders, a chance to coordinate
the release of fixes and reduce the vulnerability window. Please treat the
following information as confidential until the proposed public
disclosure date.

Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: Designate
Affects: 2015.1.0 through 1.0.0.0b1

Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max allowed DNS
packet size, an authenticated user may cause the Designate mDNS service
to enter an infinite loop, triggering a DoS.

Proposed patch:
See attached patches. Unless a flaw is discovered in them, these patches
will be merged to stable/kilo and master on the public disclosure date.

CVE: TBA
Launchpad Bug #: 1471161
RedHat Bug #: 1236014

Proposed public disclosure date/time: 2015-07-28, 1500UTC

Please do not make the issue public (or release public patches) before
this coordinated embargo date.

Regards,

--
Kiall Mac Innes
OpenStack Designate PTL

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206578

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/206579

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206580

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206581

[Kiall Mac Innes (kiall)]

Public disclosure date/time (2015-07-28, 1500UTC) reached, bug has been marked public, patches uploaded + approved, and going through gate now.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206580
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=969db41bf6d4c14d9925b801c36685e6b7d3f246
Submitter: Jenkins
Branch: stable/kilo

commit 969db41bf6d4c14d9925b801c36685e6b7d3f246
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100

    Ensure a single RRSet over max_packet_size doesn't loop forever

    A single RRSet exceeding the max_packet_size configuration option would
    result in mDNS entering a infinite loop, causing a Denial of Service.

    Change-Id: Icab7cfe18a0e46cf3ec5d7087eb878e9d8653a4b
    Closes-Bug: 1471161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206578
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=dac3fbc1f78dcd49dadebbd990db7b2d1a68e8c0
Submitter: Jenkins
Branch: master

commit dac3fbc1f78dcd49dadebbd990db7b2d1a68e8c0
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100

    Ensure a single RRSet over max_packet_size doesn't loop forever

    A single RRSet exceeding the max_packet_size configuration option would
    result in mDNS entering a infinite loop, causing a Denial of Service.

    Change-Id: Icab7cfe18a0e46cf3ec5d7087eb878e9d8653a4b
    Closes-Bug: 1471161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206581
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=46f187f53506de1237cdd14b5d2a69e73fd8f720
Submitter: Jenkins
Branch: stable/kilo

commit 46f187f53506de1237cdd14b5d2a69e73fd8f720
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100

    Ensure RecordSet quotas are enforced

    Implement RecordSet's per domain, and Records per RecordSet
    quota enforcement.

    Change-Id: If7afc70cd1ebe2e18864859de51c6ccd15c6a43c
    Partial-Bug: 1471161

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/206579
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=789709f1db081222c2e4d21d287781e437b76a95
Submitter: Jenkins
Branch: master

commit 789709f1db081222c2e4d21d287781e437b76a95
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100

    Ensure RecordSet quotas are enforced

    Implement RecordSet's per domain, and Records per RecordSet
    quota enforcement.

    Change-Id: If7afc70cd1ebe2e18864859de51c6ccd15c6a43c
    Partial-Bug: 1471161

[Kiall Mac Innes (kiall)]

Two CVE numbers have been assigned to track the two parts of this:

CVE-2015-5694 and CVE-2015-5695

https://security-tracker.debian.org/tracker/CVE-2015-5694
https://security-tracker.debian.org/tracker/CVE-2015-5695

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Master Branch Patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]