systemd-resolved tries to mitigate DVE-2018-0001 even if DNSSEC=yes

Bug #1796501 reported by jrb0001 on 2018-10-06
38
This bug affects 5 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Medium
Balint Reczey
Bionic
Medium
Dimitri John Ledkov
Cosmic
Medium
Dimitri John Ledkov
Disco
Medium
Dimitri John Ledkov

Bug Description

[impact]

an NXDOMAIN response from a dns server when systemd-resolved is configured as DNSSEC=yes breaks dns resolution as it downgrades from DNSSEC.

[test case]

see comment 9

[regression potential]

as with the original patch that introduced this problem, this has the potential to break dns resolution.

[other info]

original description:

I ask systemd-resolved through dig to resolve the SOA of test.asdf. (doesn't exist) but it returns SERVFAIL instead of NXDOMAIN. It seems to do the following steps:
1. Ask upstream for SOA of test.asdf. with EDNS0, DO-bit and 4k size.
2. Ask upstream for SOA of test.asdf. with EDNS0 and DO-bit.
3. Ask upstream for SOA of test.asdf. with EDNS0.
4. Ask upstream for SOA of test.asdf. without EDNS0.
5. Repeat 1-4 for DS of test.asdf.
6. Repeat 1-5 for asdf.
7. Ask upstream for SOA of . with EDNS0, DO-bit and 4k size.
8. Ask upstream for DNSKEY of . with EDNS0, DO-bit and 4k size.

The upstream returns an unfragmented NXDOMAIN response for steps 1-6, an unfragmented NOERROR response for step 7 and a fragmented NOERROR response for step 8 which is the correct behaviour. DNSSEC records are included in the response if the DO-bit in the request was set.

systemd-resolved should take the response from step 1 and start with validation instead of starting useless retries with reduced feture set. Step 3 and 4 are completely useless and probably lead to the SERVFAIL because I have configured it with DNSSEC=yes to prevent downgrade attacks.

This regression seems to be caused by the patch resolved-Mitigate-DVE-2018-0001-by-retrying-NXDOMAIN-with.patch. The downgrade logic should only be executed if it is configured as DNSSEC=allow-downgrade or DNSSEC=no. See also https://github.com/systemd/systemd/pull/8608#issuecomment-396927885.

jrb0001 (jrb0001) wrote :
jrb0001 (jrb0001) wrote :
Bryan Quigley (bryanquigley) wrote :

Can this bug make the complete failure of DNS (like https://github.com/systemd/systemd/issues/6490) more likely?

If SERVFAIL is for the DNS server, that sounds like this would cause more failures of DNS per the other issue.

tags: added: sts
jrb0001 (jrb0001) wrote :

I think the downgrade behaviour of systemd-resolved is the same as in that upstream bug although it is triggered differently. Except that it only breaks that single NXDOMAIN query in my case while it sounds like a permanent failure in the upstream bug.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Eric Desrochers (slashd) on 2019-02-19
Changed in systemd (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
status: Confirmed → In Progress
importance: Undecided → Medium
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in systemd (Ubuntu Cosmic):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Dan Streetman (ddstreet) on 2019-04-08
tags: removed: sts
jrb0001 (jrb0001) wrote :

Has there been any progress so far? systemd-resolved would be nice in theory, but not if it breaks half of the websites like it did before I stopped using it.

Dan Streetman (ddstreet) on 2019-05-31
tags: added: sts
Bryan Quigley (bryanquigley) wrote :

A simple case on Disco+ that I believe is related to the DVE workaround is:
resolvectl query www.engadget.com

DNSSEC doesn't appear to actually be involved on the domains. but with DNSSEC=(not yes) it works.

Bryan Quigley (bryanquigley) wrote :

I grabbed the top 500 hosts in an Eaon LXD container with DNS=1.1.1.1
wget -O top500.csv https://moz.com/top-500/download/?table=top500Domains
cut -d, -f2 < top500.csv | cut -d\" -f2 > top500

I ran this script twice (with and without dnssec=yes):
while read p; do
  sleep 1
  echo "$p"
  resolvectl query $p > with_dnssec/$p
done <top500

The following domains failed only with DNSSEC=yes (and all failures included DVE- notices in journal).
people.com.cn
search.yahoo.com
news.yahoo.com

(oddly engadget wasn't on the list.. There may be a difference between netword/network-manager?)

Bryan Quigley (bryanquigley) wrote :

I've confirmed, that #9 is is not reproducible with systemd from Debian. The runs from there with our without DNSSEC=yes are the same. They differ on Ubuntu.

Bryan Quigley (bryanquigley) wrote :

I just built a package that just reverts it for Bionic and Disco : https://launchpad.net/~bryanquigley/+archive/ubuntu/1796501

Will confirm results tomorrow but so far with DNSSEC=yes:

Bionic with DVE-2018-0001 patch: Can't resolve europa.eu
Bionic with patch reverted: Can resolve europa.eu

Disco with DVE-2018-0001 patch: Can't resolve people.com.cn, search.yahoo.com, news.yahoo.com
Disco with patch reverted: Can resolve those three domains.

Changed in systemd (Ubuntu Cosmic):
status: In Progress → Won't Fix
Bryan Quigley (bryanquigley) wrote :

I've confirmed the fix causes other issues (above), and is still needed for wifi to work at Starbucks (In Toronto) - although DNS failed after accepting to east.datavalet.io.

I'm going to reach out to Datavalet and see if they have any thoughts.

Bryan Quigley (bryanquigley) wrote :

@xnox Your updated patch (https://github.com/systemd/systemd/commit/50b9974aee29efb8118a20360b0d521f58110afd) on the GH issue fixes this issue AFAICT. Can we have the patch updated in Ubuntu?

I'm happy to make a debdiff.. but it is all your patches..

Bryan Quigley (bryanquigley) wrote :

Built a eoan package with xnox's updated patch in my ppa: https://launchpad.net/~bryanquigley/+archive/ubuntu/1796501/+packages

1. Confirm failure with DNSSEC=yes, DNS server 1.1.1.1
$ resolvectl query people.com.cn
people.com.cn: resolve call failed: DNSSEC validation failed: failed-auxiliary

2. Add PPA and upgrade systemd to PPA version.

3. Confirm success:
resolvectl query people.com.cn
people.com.cn: 106.48.12.140 -- link: ens2
               106.48.12.141 -- link: ens2
               (hpcc-download-foreign.chinacache.net)

The attachment "eoan debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Bryan Quigley (bryanquigley) wrote :

I've confirmed that with this update patch my issue in C#9 is indeed fixed. It also should be more performant for non-DNSSEC=yes users.

Dimitri John Ledkov (xnox) wrote :

@bryanquigley are you going to SRU that?

And please just that alone?

Bryan Quigley (bryanquigley) wrote :

Yes, if we can get it into dev, I'd happily make debdiffs to SRU it to bionic/disco.

>And please just that alone?
Yes, just updating the patch to your latest version. I'm ok if it needs to be queued up for SRU with other systemd changes if that's what you are getting at.

tags: added: id-5d92536b4bcd9c68caddc01c
Victor Tapia (vtapia) wrote :
Balint Reczey (rbalint) wrote :

I'm adding this patch to the next upload I'm preparing in https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3797/+packages .

Changed in systemd (Ubuntu):
assignee: Dimitri John Ledkov (xnox) → Balint Reczey (rbalint)
Eric Desrochers (slashd) on 2019-10-04
tags: added: sts-sponsor-ddstreet
Dan Streetman (ddstreet) on 2019-10-07
tags: added: ddstreet systemd
tags: added: bionic disco
Dan Streetman (ddstreet) on 2019-10-10
description: updated
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package systemd - 242-7ubuntu2

---------------
systemd (242-7ubuntu2) eoan; urgency=medium

  [ Bryan Quigley ]
  * Update patch for resolved: Mitigate DVE-2018-0001, by retrying NXDOMAIN
    without EDNS0. This disables the workaround if DNSSEC=yes.
    Falls back directly to simple UDP instead of trying an intermediate.
    (LP: #1796501)
    Author: Bryan Quigley
    File: debian/patches/resolved-Mitigate-DVE-2018-0001-by-retrying-NXDOMAIN-with.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=2974114ed9b89ea922a23893e8eff70d5cac77fe

  [ Balint Reczey ]
  * Pass personality test even when i386 userland runs on amd64 kernel
    File: debian/patches/debian/UBUNTU-test-Pass-personality-test-even-when-i386-userland-runs-o.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=42e0bfc426f19430f6768ef4922a9531a345765f
  * Refresh patches
    Files:
    - debian/patches/Revert-namespace-be-more-careful-when-handling-namespacin.patch
    - debian/patches/debian/Ubuntu-core-in-execute-soft-fail-setting-Nice-priority-when.patch
    - debian/patches/test-execute-Filter-dev-.lxc-in-exec-dynamicuser-statedir.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ff8387be07322230e9afe87f1c767ee241e9a0e1

 -- Balint Reczey <email address hidden> Tue, 08 Oct 2019 22:31:17 +0200

Changed in systemd (Ubuntu):
status: In Progress → Fix Released

Hello jrb0001, or anyone else affected,

Accepted systemd into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/240-6ubuntu5.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Disco):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-disco

All autopkgtests for the newly accepted systemd (240-6ubuntu5.8) for disco have finished running.
The following regressions have been reported in tests triggered by the package:

prometheus-bind-exporter/unknown (armhf)
php7.2/7.2.24-0ubuntu0.19.04.1 (armhf)
gvfs/1.40.1-1ubuntu0.1 (ppc64el)
pdns-recursor/unknown (armhf)
webhook/unknown (armhf)
munin/2.0.47-1ubuntu3 (armhf, arm64)
systemd/240-6ubuntu5.8 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/disco/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello jrb0001, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.32 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in systemd (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Bryan Quigley (bryanquigley) wrote :

Verified in disco with, ii systemd 240-6ubuntu5.8

Dnssec and not are now consistent, tested general functionality with dnssec=yes and not set as well.

tags: added: verification-done-disco
removed: verification-needed-disco
Bryan Quigley (bryanquigley) wrote :

Verified in bionic with 237-3ubuntu10.32

Dnssec and not are now consistent, tested general functionality with dnssec=yes and not set as well.

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.32) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (ppc64el)
linux/unknown (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Hello jrb0001, or anyone else affected,

Accepted systemd into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/237-3ubuntu10.33 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

tags: added: verification-needed verification-needed-bionic
removed: verification-done verification-done-bionic

All autopkgtests for the newly accepted systemd (237-3ubuntu10.33) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

gvfs/1.36.1-0ubuntu1.3.3 (ppc64el, amd64)
dovecot/1:2.2.33.2-1ubuntu4.5 (armhf)
umockdev/0.11.1-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.