Designate mDNS DoS through incorrect handling of large RecordSets
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
| Designate |
Critical
|
Kiall Mac Innes | ||
| Kilo |
Critical
|
Kiall Mac Innes | ||
| designate (Ubuntu) |
Undecided
|
Unassigned |
Bug Description
Designate does not enforce the DNS protocol limit concerning record set sizes when adding records for a domain name. The protocol limit is slightly less than 2**16 bytes because of some data that must accompany any record set, as part of the zone file transfer.
As a result, the rendering loop in desginate-mdns can does not make progress because the problematic record set at rrsets[i] will never fit (per designate/
# Render the results, yielding a packet after each TooBig exception.
i, renderer = 0, None
while i < len(rrsets):
# No renderer? Build one
if renderer is None:
…
try:
i += 1
except dns.exception.
if request.had_tsig:
…
When this happens, designate-mdns will keep sending SOA records over and over again:
sendto(37, "\0\35<
sendto(37, "\0\35<
sendto(37, "\0\35<
Here is how I triggered this. I created a domain example.org:
+------
| id | name | serial |
+------
| ded903fc-
+------
Then I added several hundred NS records for the same subdomain, using this shell command:
for x in {1..304} ; do
designate record-create --name 'sub.example.org.' --type NS --data \
$x'.xxxxxxxxxxx
ded903fc-
done
In the tested configuration, PowerDNS will attempt a zone file transfer, and trigger the bug. Because it keeps receiving data, it does not seem it will ever run into a timeout (and if it does, it will try again).
Downstream bug report: https:/
no longer affects: | designate/kilo |
Changed in designate: | |
status: | New → Triaged |
importance: | Undecided → Critical |
milestone: | none → liberty-2 |
Kiall Mac Innes (kiall) wrote : | #1 |
Kiall Mac Innes (kiall) wrote : | #2 |
Apologies, patch is based off https:/
Kiall Mac Innes (kiall) wrote : | #3 |
Finally, this fixes the DOS - Not the failure to apply quotas..
Kiall Mac Innes (kiall) wrote : | #4 |
Affects debian: https:/
Affects ubuntu: http://
RedHat doesn't seem to have an equivalent package search site, but I'm assuming since they filed this issue, it affects RedHat too.
Graham Hayes (grahamhayes) wrote : | #5 |
I think there is 2 parts to this bug:
1: Quotas were being bypassed as part of the v1 API.
2. If there was enough RRs in a RRSet MiniDNS went into a loop.
3. MiniDNS does not have a timeout.
1 is not really relevant - 2 should not happen in anycase.
The patch fixes 2, but not 3 - should we be include that in the fix?
Kiall Mac Innes (kiall) wrote : | #6 |
Kiall Mac Innes (kiall) wrote : | #8 |
Kiall Mac Innes (kiall) wrote : | #9 |
Kiall Mac Innes (kiall) wrote : | #10 |
I've gone and implemented the RRSet related quota enforcement for Master and Kilo, this was not previously implemented. I've included them in this bug, as publishing them now might indicate there's an issue with RRset sizes, but I don't consider them directly part of this DoS vulnerability.
i.e. An operator could choose to have infinite quotas (a valid use case), and we should support that.
Kiall Mac Innes (kiall) wrote : | #11 |
Attached initial OSSA yaml based on existing documents from https:/
Graham Hayes (grahamhayes) wrote : | #12 |
Updated Kilo patch
Kiall Mac Innes (kiall) wrote : | #13 |
All patches have now been validated as fixing the DoS.
Graham Hayes (grahamhayes) wrote : | #14 |
Main Patch:
Bind9 + Master - Fix Verified
Bind9 + Kilo - Fix Verified
I have not tested the Quotas patch yet, but it should be DNS Server agnostic.
Kiall Mac Innes (kiall) wrote : | #15 |
Attached reproduction script used.
Ran before applying patches, mDNS will consume 100% CPU once it's completed and zones will not propagate to the nameservers.
Ran after applying patches, mDNS will correctly propagate zones to the nameservers.
Changed in designate: | |
assignee: | nobody → Kiall Mac Innes (kiall) |
Tim Simmons (timsim) wrote : | #16 |
Verified main fix with:
Bind9 + Master
Bind9 + Kilo
Verified quotas fix on:
Master
Kilo
summary: |
- Lack of record set size checking leads to denial of service + Designate mDNS DoS through incorrect handling of large RecordSets |
Kiall Mac Innes (kiall) wrote : | #17 |
CVE # Allocation request sent, based on VMT template. Copy of email:
A vulnerability was discovered in OpenStack Designate (see below).
In order to ensure full traceability, we need a CVE number assigned
that we can attach to private and public notifications. Please treat
the following information as confidential until further public
disclosure.
Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: OpenStack Designate
Affects: OpenStack Designate version 2015.1.0
Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max
allowed DNS packet size, an authenticated user may cause the
Designate mDNS service to enter an infinite loop, triggering a DoS.
Thanks in advance,
Kiall Mac Innes
Kiall Mac Innes (kiall) wrote : | #18 |
No CVE number yet, went ahead and notified downstreams. Disclosure date set for 2015-07-28, 1500UTC.
Copy of email below:
This is an advance warning of a vulnerability discovered in OpenStack
Designate, to give you, as downstream stakeholders, a chance to coordinate
the release of fixes and reduce the vulnerability window. Please treat the
following information as confidential until the proposed public
disclosure date.
Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: Designate
Affects: 2015.1.0 through 1.0.0.0b1
Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max allowed DNS
packet size, an authenticated user may cause the Designate mDNS service
to enter an infinite loop, triggering a DoS.
Proposed patch:
See attached patches. Unless a flaw is discovered in them, these patches
will be merged to stable/kilo and master on the public disclosure date.
CVE: TBA
Launchpad Bug #: 1471161
RedHat Bug #: 1236014
Proposed public disclosure date/time: 2015-07-28, 1500UTC
Please do not make the issue public (or release public patches) before
this coordinated embargo date.
Regards,
--
Kiall Mac Innes
OpenStack Designate PTL
Changed in designate: | |
status: | Triaged → In Progress |
information type: | Private Security → Public Security |
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: master
Review: https:/
Fix proposed to branch: stable/kilo
Review: https:/
Fix proposed to branch: stable/kilo
Review: https:/
Kiall Mac Innes (kiall) wrote : | #23 |
Public disclosure date/time (2015-07-28, 1500UTC) reached, bug has been marked public, patches uploaded + approved, and going through gate now.
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/kilo
commit 969db41bf6d4c14
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100
Ensure a single RRSet over max_packet_size doesn't loop forever
A single RRSet exceeding the max_packet_size configuration option would
result in mDNS entering a infinite loop, causing a Denial of Service.
Change-Id: Icab7cfe18a0e46
Closes-Bug: 1471161
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit dac3fbc1f78dcd4
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100
Ensure a single RRSet over max_packet_size doesn't loop forever
A single RRSet exceeding the max_packet_size configuration option would
result in mDNS entering a infinite loop, causing a Denial of Service.
Change-Id: Icab7cfe18a0e46
Closes-Bug: 1471161
Changed in designate: | |
status: | In Progress → Fix Committed |
tags: | added: in-stable-kilo |
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: stable/kilo
commit 46f187f53506de1
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100
Ensure RecordSet quotas are enforced
Implement RecordSet's per domain, and Records per RecordSet
quota enforcement.
Change-Id: If7afc70cd1ebe2
Partial-Bug: 1471161
Reviewed: https:/
Committed: https:/
Submitter: Jenkins
Branch: master
commit 789709f1db08122
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100
Ensure RecordSet quotas are enforced
Implement RecordSet's per domain, and Records per RecordSet
quota enforcement.
Change-Id: If7afc70cd1ebe2
Partial-Bug: 1471161
Kiall Mac Innes (kiall) wrote : | #28 |
Two CVE numbers have been assigned to track the two parts of this:
CVE-2015-5694 and CVE-2015-5695
https:/
https:/
Changed in designate: | |
status: | Fix Committed → Fix Released |
The attachment "Master Branch Patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.
[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]
tags: | added: patch |
Changed in designate (Ubuntu): | |
status: | New → Confirmed |
Changed in designate (Ubuntu): | |
status: | Confirmed → Fix Released |
Changed in designate: | |
milestone: | liberty-2 → 1.0.0 |
Attached a patch providing an initial candidate fix, based off master.
Tests are passing, but this has not yet been validated with BIND/PowerDNS etc