systemd-resolved: DNSSEC validation failed: incompatible-server

Bug #1981431 reported by Jan-Otto Kröpke
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Fix Released
Undecided
Unassigned
Jammy
Incomplete
Undecided
Unassigned

Bug Description

Hi,

I'm running Ubuntu 22.04 using systemd 249.11-0ubuntu3.4.

2 days ago, I enabled DNSSEC=true through:

# grep DNSSEC /etc/systemd/resolved.conf.d/dnssec.conf
DNSSEC=yes

After running some hours, systemd-resolved stop working. Log lines like incompatible-server starts to spam.

Jul 09 13:51:41 htdocs systemd[1]: Starting Network Name Resolution...
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Positive Trust Anchors:
Jul 09 13:51:41 htdocs systemd-resolved[77507]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Negative trust anchors: int.creativesandbox.de
Jul 09 13:51:41 htdocs systemd-resolved[77507]: Using system hostname 'htdocs'.
Jul 09 13:51:41 htdocs systemd[1]: Started Network Name Resolution.
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 214.162.in-addr.arpa IN SOA: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 76.214.162.in-addr.arpa IN DS: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 126.76.214.162.in-addr.arpa IN DS: no-signature
Jul 09 15:40:20 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 126.76.214.162.in-addr.arpa IN SOA: no-signature
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question . IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question de IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question your-server.de IN SOA: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question clients.your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question 201.138.clients.your-server.de IN DS: incompatible-server
Jul 10 03:16:18 htdocs systemd-resolved[77507]: DNSSEC validation failed for question static.237.11.201.138.clients.your-server.de IN AAAA: incompatible-server

Mention here, I'm running multiple machines with the same config against the same upstream DNS server. From time to time, only one instance is stop working here.

Running a manual query also fails here, for example:

# resolvectl query noc3.wordfence.com
noc3.wordfence.com: resolve call failed: DNSSEC validation failed: incompatible-server

Running 'resolvectl reset-server-features' helps here and can be considered as workaround.

# resolvectl query noc3.wordfence.com
noc3.wordfence.com: resolve call failed: DNSSEC validation failed: incompatible-server
# resolvectl reset-server-features
# resolvectl query noc3.wordfence.com
noc3.wordfence.com: 35.155.126.231 -- link: eth0

-- Information acquired via protocol DNS in 26.5ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

By reading issues upstream looks like https://github.com/systemd/systemd/issues/6490.

A fix is implemented (https://github.com/systemd/systemd/pull/18624) and released in 248 which is included in Ubuntu 22.04.

But there is another fix around this issue (https://github.com/systemd/systemd/pull/20214) which is released in systemd 250.

I would like to know if it's possible to backport this fix into Ubuntu 22.04.

Thanks.

description: updated
Nick Rosbrook (enr0n)
tags: added: rls-jj-incoming
Revision history for this message
Lukas Märdian (slyon) wrote :

The fix is already included in systemd >= v250 (i.e. Kinetic+)

tags: added: fr-2550
Changed in systemd (Ubuntu):
status: New → Fix Released
tags: removed: rls-jj-incoming
Revision history for this message
Nick Rosbrook (enr0n) wrote :

> But there is another fix around this issue (https://github.com/systemd/systemd/pull/20214) which is released in systemd 250.

Taking a closer look, this was actually backported to 249.4 [1] and is present in Jammy, so this appears to be a similar but different issue.

[1] https://github.com/systemd/systemd-stable/commit/8280bec34df8e35592f4a4a549127471a9199231

Revision history for this message
Nick Rosbrook (enr0n) wrote :

If you have not already, I would suggest opening a new upstream issue. After a quick look through https://github.com/systemd/systemd/issues, it seems there have been many closely related DNSSEC issues, so it is not obvious to me if there is a patch for your issue.

Revision history for this message
Jan-Otto Kröpke (jokroepke) wrote :

I'm able to assist here, e.g. by enable debugging logs. I need some instructions what I should setup and have to provide here.

Nick Rosbrook (enr0n)
Changed in systemd (Ubuntu Jammy):
status: New → Incomplete
Revision history for this message
Nick Rosbrook (enr0n) wrote (last edit ):

That would be helpful. You can turn on debug-level logging for systemd-resolved by running `systemctl edit systemd-resolved`, and add the following line to the [Service] section:

Environment=SYSTEMD_LOG_LEVEL=debug

Then `systemctl restart systemd-resolved`. When you have some debug logs from an occurrence of this issue, I can take another look.

Revision history for this message
Jan-Otto Kröpke (jokroepke) wrote :

Thanks for your answer.

In mean time, I setup 2 additional lxd containers, one running jammy, one running kinetic and I'm hit the error again on kinetic by running dig +dnssec on random DNS names in a loop

I follow your instructions, but I used `Environment=SYSTEMD_LOG_LEVEL=debug` to gain log messages.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

Whoops, edited my comment above to fix that.

Revision history for this message
Jan-Otto Kröpke (jokroepke) wrote :
Revision history for this message
Jan-Otto Kröpke (jokroepke) wrote :

Debug log of systemd-networkd

Shell script to reproduce it:

curl https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-random-domains.txt -O
curl https://raw.githubusercontent.com/opendns/public-domain-lists/master/opendns-top-domains.txt -L >> opendns-random-domains.txt

for i in $(cat opendns-random-domains.txt); do sleep 1; dig +short +dnssec $i @127.0.0.53; done

Logs are from 20:00:00 to 00:07:55. Error appeared on 00:07:53.

The error comes from kinetic lxd container again, while the jammy lxd still running. Upstream DNS Server is dnsmasq (from LXD).

Revision history for this message
Jan-Otto Kröpke (jokroepke) wrote :

One more log.

I also test different upstream DNS server (hetzner.de; Google DNS) but I'm not sure what is wrong here.

Revision history for this message
Nick Rosbrook (enr0n) wrote :

It looks like jokroepke created an upstream issue: https://github.com/systemd/systemd/issues/24098.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.