systemd-resolved tries to mitigate DVE-2018-0001 even if DNSSEC=yes

Bug #1796501 reported by jrb0001 on 2018-10-06
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Medium
Dimitri John Ledkov
Bionic
Medium
Dimitri John Ledkov
Cosmic
Medium
Dimitri John Ledkov
Disco
Medium
Dimitri John Ledkov

Bug Description

I ask systemd-resolved through dig to resolve the SOA of test.asdf. (doesn't exist) but it returns SERVFAIL instead of NXDOMAIN. It seems to do the following steps:
1. Ask upstream for SOA of test.asdf. with EDNS0, DO-bit and 4k size.
2. Ask upstream for SOA of test.asdf. with EDNS0 and DO-bit.
3. Ask upstream for SOA of test.asdf. with EDNS0.
4. Ask upstream for SOA of test.asdf. without EDNS0.
5. Repeat 1-4 for DS of test.asdf.
6. Repeat 1-5 for asdf.
7. Ask upstream for SOA of . with EDNS0, DO-bit and 4k size.
8. Ask upstream for DNSKEY of . with EDNS0, DO-bit and 4k size.

The upstream returns an unfragmented NXDOMAIN response for steps 1-6, an unfragmented NOERROR response for step 7 and a fragmented NOERROR response for step 8 which is the correct behaviour. DNSSEC records are included in the response if the DO-bit in the request was set.

systemd-resolved should take the response from step 1 and start with validation instead of starting useless retries with reduced feture set. Step 3 and 4 are completely useless and probably lead to the SERVFAIL because I have configured it with DNSSEC=yes to prevent downgrade attacks.

This regression seems to be caused by the patch resolved-Mitigate-DVE-2018-0001-by-retrying-NXDOMAIN-with.patch. The downgrade logic should only be executed if it is configured as DNSSEC=allow-downgrade or DNSSEC=no. See also https://github.com/systemd/systemd/pull/8608#issuecomment-396927885.

jrb0001 (jrb0001) wrote :
jrb0001 (jrb0001) wrote :
Bryan Quigley (bryanquigley) wrote :

Can this bug make the complete failure of DNS (like https://github.com/systemd/systemd/issues/6490) more likely?

If SERVFAIL is for the DNS server, that sounds like this would cause more failures of DNS per the other issue.

tags: added: sts
jrb0001 (jrb0001) wrote :

I think the downgrade behaviour of systemd-resolved is the same as in that upstream bug although it is triggered differently. Except that it only breaks that single NXDOMAIN query in my case while it sounds like a permanent failure in the upstream bug.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in systemd (Ubuntu):
status: New → Confirmed
Eric Desrochers (slashd) on 2019-02-19
Changed in systemd (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
status: Confirmed → In Progress
importance: Undecided → Medium
Changed in systemd (Ubuntu Cosmic):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in systemd (Ubuntu Bionic):
assignee: nobody → Dimitri John Ledkov (xnox)
Changed in systemd (Ubuntu Cosmic):
status: New → In Progress
Changed in systemd (Ubuntu Bionic):
status: New → In Progress
Changed in systemd (Ubuntu Cosmic):
importance: Undecided → Medium
Changed in systemd (Ubuntu Bionic):
importance: Undecided → Medium
Dan Streetman (ddstreet) on 2019-04-08
tags: removed: sts
jrb0001 (jrb0001) wrote :

Has there been any progress so far? systemd-resolved would be nice in theory, but not if it breaks half of the websites like it did before I stopped using it.

Dan Streetman (ddstreet) on 2019-05-31
tags: added: sts
Bryan Quigley (bryanquigley) wrote :

A simple case on Disco+ that I believe is related to the DVE workaround is:
resolvectl query www.engadget.com

DNSSEC doesn't appear to actually be involved on the domains. but with DNSSEC=(not yes) it works.

Bryan Quigley (bryanquigley) wrote :

I grabbed the top 500 hosts in an Eaon LXD container with DNS=1.1.1.1
wget -O top500.csv https://moz.com/top-500/download/?table=top500Domains
cut -d, -f2 < top500.csv | cut -d\" -f2 > top500

I ran this script twice (with and without dnssec=yes):
while read p; do
  sleep 1
  echo "$p"
  resolvectl query $p > with_dnssec/$p
done <top500

The following domains failed only with DNSSEC=yes (and all failures included DVE- notices in journal).
people.com.cn
search.yahoo.com
news.yahoo.com

(oddly engadget wasn't on the list.. There may be a difference between netword/network-manager?)

Bryan Quigley (bryanquigley) wrote :

I've confirmed, that #9 is is not reproducible with systemd from Debian. The runs from there with our without DNSSEC=yes are the same. They differ on Ubuntu.

Bryan Quigley (bryanquigley) wrote :

I just built a package that just reverts it for Bionic and Disco : https://launchpad.net/~bryanquigley/+archive/ubuntu/1796501

Will confirm results tomorrow but so far with DNSSEC=yes:

Bionic with DVE-2018-0001 patch: Can't resolve europa.eu
Bionic with patch reverted: Can resolve europa.eu

Disco with DVE-2018-0001 patch: Can't resolve people.com.cn, search.yahoo.com, news.yahoo.com
Disco with patch reverted: Can resolve those three domains.

Changed in systemd (Ubuntu Cosmic):
status: In Progress → Won't Fix
Bryan Quigley (bryanquigley) wrote :

I've confirmed the fix causes other issues (above), and is still needed for wifi to work at Starbucks (In Toronto) - although DNS failed after accepting to east.datavalet.io.

I'm going to reach out to Datavalet and see if they have any thoughts.

Bryan Quigley (bryanquigley) wrote :

@xnox Your updated patch (https://github.com/systemd/systemd/commit/50b9974aee29efb8118a20360b0d521f58110afd) on the GH issue fixes this issue AFAICT. Can we have the patch updated in Ubuntu?

I'm happy to make a debdiff.. but it is all your patches..

Bryan Quigley (bryanquigley) wrote :

Built a eoan package with xnox's updated patch in my ppa: https://launchpad.net/~bryanquigley/+archive/ubuntu/1796501/+packages

1. Confirm failure with DNSSEC=yes, DNS server 1.1.1.1
$ resolvectl query people.com.cn
people.com.cn: resolve call failed: DNSSEC validation failed: failed-auxiliary

2. Add PPA and upgrade systemd to PPA version.

3. Confirm success:
resolvectl query people.com.cn
people.com.cn: 106.48.12.140 -- link: ens2
               106.48.12.141 -- link: ens2
               (hpcc-download-foreign.chinacache.net)

The attachment "eoan debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.