DNS server capability detection is broken and has fatal consequences to resolving when DNSSEC is enabled

[Avamander]

I'm running Ubuntu 19.10

I'm on latest version available from repositories, systemd 242

I'm expecting upstream DNS server capabilities being detected correctly and DNSSEC to keep working. Alternatively I'd expect a method of disabling capability checks instead of DNSSEC.

Currently instead resolved misdetect features suddenly, stops resolving all together (fails closed, which is somewhat good). Capability reset is a very temporary fix.

A suggested fix could be (ordered based on how nice of a solution it is):

a. The capability detection is fixed (https://github.com/systemd/systemd/issues/9384)

b. Force-disabling capability detection exists (this is what I also requested here: https://github.com/systemd/systemd/issues/14435)

c. Patch Ubuntu version not to allow such a foot gun, update documentation (this is theoretically what Ubuntu could do meanwhile)

d. Remove DNSSEC from resolved

[Dan Streetman (ddstreet)]

Can you post logs from when the capability mis-detection happens? What indication do you have that is what's happening? How do you have DNSSEC configured?

[Avamander (avamander)]

Yes, DNSSEC is configured.

Logs say this:
```
Using degraded feature set (UDP+EDNS0+DO) for DNS server 192.168.1.1.
```
and then it starts to spam lines like this:
```
DNSSEC validation failed for question internetsociety.org IN A: incompatible-server
```

[Avamander (avamander)]

Removed the link to a separate issue.

[Dan Streetman (ddstreet)]

> Yes, DNSSEC is configured.

HOW do you have DNSSEC configured. Not a yes/no question.

> Logs say this:

please include more than that; single lines don't help debug. Attach the entire syslog if you're unsure how much to paste in.

Also please paste/attach the output of:

$ systemd-resolve --status --no-pager

and

$ journalctl --no-pager -b -u systemd-resolved

I'm specifically looking for lines like this:
"Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001"

[Avamander (avamander)]

> HOW do you have DNSSEC configured. Not a yes/no question.

Actually it's exactly a "yes" in the configuration file.

[Avamander (avamander)]

I will need to wait for the bug to trigger again, logs have rotated since the last time I had `yes` in the config. Also, those "single line"s are actually what I did see repeatedly until I had to stop and reconfigure resolved because it made internet usage impossible.

[Avamander (avamander)]

```
systemd[1]: Starting Network Name Resolution...
systemd-resolved[1392]: Positive Trust Anchors:
systemd-resolved[1392]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
systemd-resolved[1392]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
systemd-resolved[1392]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 2
systemd-resolved[1392]: Using system hostname 'machine'.
systemd[1]: Started Network Name Resolution.
systemd-resolved[1392]: DNSSEC validation failed for question 0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question b.9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question b.9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 1.b.9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN DS: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question 1.b.9.c.c.3.2.2.0.a.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa IN SOA: no-signature
systemd-resolved[1392]: DNSSEC validation failed for question d...

[Halvor Lyche Strandvoll (halvors)]

Experiencing this on Ubuntu 20.10 as well.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue