systemd local DNS tests failing with timeout

[Stefan Bader]

Investigations done in 22.04/Jammy but may be affecting other series, too.

The dnsmasq package recently was updated from 2.86-1.1ubuntu0.5 to 2.90-0ubuntu0.22.04.1. This seems to have brought back the same issue reported in bug #1957086. Sounds like both have interaction issues.

To reproduce:

$ pull-lp-source systemd jammy
# Install test deps
$ sudo apt install systemd udev libpam-systemd libnss-systemd acl locales evemu-tools python3 pkg-config cryptsetup-bin systemd-sysv policykit-1 dnsmasq-base
$ cd systemd-249.11/test/
$ sudo ./networkd-test.py

======================================================================
ERROR: test_resolved_domain_restricted_dns (__main__.DnsmasqClientTest)
resolved: domain-restricted DNS servers
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/systemd-249.11/test/./networkd-test.py", line 678, in test_resolved_domain_restricted_dns
    out = subprocess.check_output(['resolvectl', 'query', 'math.lab'])
  File "/usr/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['resolvectl', 'query', 'math.lab']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 35 tests in 252.167s

[Marc Deslauriers (mdeslaur)]

The same issue was present with the old dnsmasq package...for example:

https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/s/systemd/20240224_133847_88f29@/log.gz

[Stefan Bader (smb)]

Sounds like some interaction issue between the system, systemd, and dnsmasq somehow. As this fails on and off since a very long time. When I test with the older dnsmasq and getting a success the time spent on the test seems overly long. Like there is something not quite right even then.

[Marc Deslauriers (mdeslaur)]

It appears most of the systemd autopkgtest failures are because of this flaky test:

https://autopkgtest.ubuntu.com/packages/systemd/jammy/amd64

The effort required to manually retrigger systemd autopkgtests because of that flaky test is substantial. We should disable that particular test unless someone manages to fix it properly.

[Lukas Märdian (slyon)]

Could you please provide journalctl debug logs from systemd-resolved, so we can get more details of what's going on?

e.g. put this into the systemd-resolved override.conf

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

$ sudo systemctl edit systemd-resolved
$ sudo systemctl restart systemd-resolved

And maybe also provide dnsmasq logs.