gdnsd fails to start, as it tries to listen to [::]:53

[Dimitri John Ledkov]

gdnsd by default tries to bind to any address on port 53

which is taken by systemd-resolved now to fix https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1731522

Well, specifically systemd-resolved takes upd/tcp 127.0.0.53:53, which is in conflict with tcp/udp [::]:53 of gdnsd.

I don't believe there exists packaging integration with gdnsd, like there is for dnsmasq to ignore / skip and not bind to certain interfaces.

What should be done to fix gdnsd? Ship a config file, or a compiled in default, to listen on localhost upd only? or to ship config snippet to override systemd-resolved behaviour to not bind on tcp?

[Dimitri John Ledkov (xnox)]

The current autopkgtest of gdnsd probably needs to be badtested too.

[Steve Langasek (vorlon)]

It is not reasonable for gdnsd to cause degradation of the system's local resolver when installed; we should therefore not ship a config snippet that overrides systemd-resolved behavior.

Other DNS servers, such as bind9, will by default bind separately to each interface, and are therefore unaffected by this conflict.

gdnsd should do similar, in order to not conflict with other dns servers bound to other addresses.

I think a reasonable default behavior for the gdnsd package today would be for it to bind to localhost only (127.0.0.1 + ::1) to allow it to be installable by default, and let users configure different addresses to listen to as they prefer.

I do not plan to implement this in the near term; unassigning myself.

[Faidon Liambotis (paravoid)]

(Debian maintainer for gdnsd here)

Since this bug was filed, gdnsd acquired autopkgtests, that among other things test that the package works in a clean container. In Debian, it does, and 3.5.2-1 has migrated to bullseye and will be part of that release. In Ubuntu, it does not because of this bug, which has blocked the migration of 3.x versions to hirsute. I was asking in #ubuntu-release earlier today about how to handle this, suggesting to either remove the package from Ubuntu, or force-badtest this; a lot of ideas were mentioned, but I don't think we came into any conclusion.

It is worth noting that this is not a new issue; 2.4.2-1 (in hirsute) is already affected and fails to install by default on Ubuntu systems. autopkgtests that were added in 3.5.2-1 just reveal this underlying issue. (Thanks to Colin Watson for actually testing this on a hirsute LXC).

There is a lot to unpack here:

* This behavior is triggered by Ubuntu shipping with a resolver (systemd-resolved, AIUI) listening on 127.0.0.53:53 on the default install. (This is not the case in Debian, so this issue is Ubuntu-specific). I suspect this is one of a class of issues of differing default installs breaking unrelated packages, but more pronounced in this case because a) this package has autopkgtests that test that it actually works :) b) the conflict is with a very common default install (all Ubuntu systems, AIUI). I haven't checked other (authoritative) DNS servers, but I did mention on IRC a random example of how "apt install tinysshd" would fail on an Ubuntu install that has openssh-server installed, which from what I gathered is the case by default in Ubuntu cloud images (cloud images being a much more limited example, though).

* An authoritative server listening on localhost is not a very helpful default configuration as it's unlikely to be used that way. Standard practice in Debian systems has been that "apt install xyz" results in a package that's useful out of the box, and for daemons, we've interpreted that as "listens by default to the network with a semi-working config", and have for years applying that to the archive. If one installs nginx or apache2, they expect it to listen on :::80. Moreover, no real attempts have been made at the distribution level of dealing with port conflicts in the same way we've dealt with filesystem conflicts; if one installs apache2 after they've installed nginx, apache2.service will fail. Some packages do have code to handle this (e.g. in the example above, Dropbear disables itself if it finds OpenSSH on the system) but it's not something standardized at the distribution level. So I hope this explains a bit why I've made the decision in the first place for the gdnsd package to start by default and listen by :::53 in Debian systems (also the upstream default by the way).

* As a side-note: a particular complexity with DNS servers is that traditionally port 53 has been shared by resolvers and authoritative servers, with daemons like bind9 supporting both. This has changed in the modern Internet, with admins expected to run separate configs for each purpose, and new generation servers having even separate daemons/codebases for each pu...

[Steve Langasek (vorlon)]

Thanks (belatedly) for your comment, Faidon. I recognize that this is a complicated issue to resolve. However, at its root, I am unwilling to sign off on a package whose installation will cause dpkg and apt to exit non-zero on a default Ubuntu system. There are various ways this could be addressed; the package could bind 127.0.0.1 by default, or it could not autostart the service, or something else. But the package has been blocked in -proposed now for 3 years because of this issue, preventing any updates - including possible security updates of a security-sensitive package (a nameserver). So until this is resolved in some manner, I think the best course of action here is to remove the package, from both the release pocket and the -proposed pocket.

Leaving this bug report open, as a breadcrumb if/when there are further syncs of the package that have the same issue.

[Faidon Liambotis (paravoid)]

So fast forward a couple of years:

deb-systemd-invoke doesn't fail these days if the systemd service cannot start -- so the package installation will work and "apt install" will complete successfully. The systemd unit will be marked as failed, in the default configuration of Ubuntu systems where systemd-resolved is running. This conflict is fundamental, and the system administrator will have to resolve it anyway if they decide to run an DNS server (any, not just gdnsd) on their Ubuntu system.

Furthermore, I've now modified the autopkgtests to be skippable in case the daemon cannot start.

Right now the conditions for skipping are kinda barebones, bailing out on the tests if pidof says the daemon isn't running. (One of the tests, the upstream testsuite one, will still run, though as that runs over another instance and port). In the future (post-freeze) I'll try to think of doing something smarter like detecting whether another daemon is using the socket, and perhaps even stopping it (w/ breaks-testbed) to properly test gdnsd instead. I know it sounds ugly, but this is an AuthDNS server and thus meant to run on port 53 -- IMHO proper CI means testing it under the conditions that it's going to be run in production.

But in the meantime, I think we can finally resolve this bug!