systemd-resolved: DNSSEC validation failed: incompatible-server

[Jan-Otto Kröpke]

Hi,

I'm running Ubuntu 22.04 using systemd 249.11-0ubuntu3.4.

2 days ago, I enabled DNSSEC=true through:

# grep DNSSEC /etc/systemd/resolved.conf.d/dnssec.conf
DNSSEC=yes

After running some hours, systemd-resolved stop working. Log lines like incompatible-server starts to spam.

Jul 09 13:51:41 htdocs systemd[1]: Starting Network Name Resolution...
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Positive Trust Anchors:
Jul 09 13:51:41 htdocs systemd-resolved[77507]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Negative trust anchors: int.creativesandbox.de
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Using system hostname 'htdocs'.
Jul 09 13:51:41 htdocs systemd[1]: Started Network Name Resolution.
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 214.162.in-addr.arpa IN SOA: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 76.214.162.in-addr.arpa IN DS: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 126.76.214.162.in-addr.arpa IN DS: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 126.76.214.162.in-addr.arpa IN SOA: no-signature
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question . IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question de IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question your-server.de IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question clients.your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 201.138.clients.your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question static.237.11.201.138.clients.your-server.de IN AAAA: incompatible-server

Mention here, I'm running multiple machines with the same config against the same upstream DNS server. From time to time, only one instance is stop working here.

Running a manual query also fails here, for example:

# resolvectl query noc3.wordfence.com
noc3.wordfence.com: resolve call failed: DNSSEC validation failed: incompatible-server

Running 'resolvectl reset-server-features' helps here and can be considered as workaround.

# resolvectl query noc3.wordfence.com
noc3.wordfence.com: resolve call failed: DNSSEC validation failed: incompatible-server
# resolvectl reset-server-features
# resolvectl query noc3.wordfence.com
noc3.wordfence.com: 35.155.126.231 -- link: eth0

-- Information acquired via protocol DNS in 26.5ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

By reading issues upstream looks like https://github.com/systemd/systemd/issues/6490.

A fix is implemented (https://github.com/systemd/systemd/pull/18624) and released in 248 which is included in Ubuntu 22.04.

But there is another fix around this issue (https://github.com/systemd/systemd/pull/20214) which is released in systemd 250.

I would like to know if it's possible to backport this fix into Ubuntu 22.04.

Thanks.

[Lukas Märdian (slyon)]

The fix is already included in systemd >= v250 (i.e. Kinetic+)

[Nick Rosbrook (enr0n)]

> But there is another fix around this issue (https://github.com/systemd/systemd/pull/20214) which is released in systemd 250.

Taking a closer look, this was actually backported to 249.4 [1] and is present in Jammy, so this appears to be a similar but different issue.

[1] https://github.com/systemd/systemd-stable/commit/8280bec34df8e35592f4a4a549127471a9199231

[Nick Rosbrook (enr0n)]

If you have not already, I would suggest opening a new upstream issue. After a quick look through https://github.com/systemd/systemd/issues, it seems there have been many closely related DNSSEC issues, so it is not obvious to me if there is a patch for your issue.

[Jan-Otto Kröpke (jokroepke)]

I'm able to assist here, e.g. by enable debugging logs. I need some instructions what I should setup and have to provide here.

[Nick Rosbrook (enr0n)]

That would be helpful. You can turn on debug-level logging for systemd-resolved by running `systemctl edit systemd-resolved`, and add the following line to the [Service] section:

Environment=SYSTEMD_LOG_LEVEL=debug

Then `systemctl restart systemd-resolved`. When you have some debug logs from an occurrence of this issue, I can take another look.

[Jan-Otto Kröpke (jokroepke)]

Thanks for your answer.

In mean time, I setup 2 additional lxd containers, one running jammy, one running kinetic and I'm hit the error again on kinetic by running dig +dnssec on random DNS names in a loop

I follow your instructions, but I used `Environment=SYSTEMD_LOG_LEVEL=debug` to gain log messages.

[Nick Rosbrook (enr0n)]

Whoops, edited my comment above to fix that.

[Jan-Otto Kröpke (jokroepke)]

Debug log of systemd-networkd

Shell script to reproduce it:

curl https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-random-domains.txt -O
curl https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-top-domains.txt -L >> opendns-random-domains.txt

for i in $(cat opendns-random-domains.txt); do sleep 1; dig +short +dnssec $i @127.0.0.53; done

Logs are from 20:00:00 to 00:07:55. Error appeared on 00:07:53.

The error comes from kinetic lxd container again, while the jammy lxd still running. Upstream DNS Server is dnsmasq (from LXD).

[Jan-Otto Kröpke (jokroepke)]

One more log.

I also test different upstream DNS server (hetzner.de; Google DNS) but I'm not sure what is wrong here.

[Nick Rosbrook (enr0n)]

It looks like jokroepke created an upstream issue: https://github.com/systemd/systemd/issues/24098.