Disabling systemd-resolved breaks dhclient resolvconf integration

[GeekSmith]

To reproduce, mask resolved:
sudo systemctl mask systemd-resolved.service

...then disable network-manager for ifupdown interfaces:
$cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile
dns=default
rc-manager=resolvconf

[ifupdown]
managed=false

[device]
wifi.scan-rand-mac-address=no

...and reboot.

You'll note that resolvconf integration with dhclient is now broken. Interfaces listed in /etc/network/interfaces or /etc/network/interfaces.d/* will not provide DNS configuration in /etc/resolv.conf and /run/resolvconf/interfaces/.

This is because /etc/dhcp/dhclient-enter-hooks.d/resolvconf defines "make_resolv_conf()" as a valid function for the BOUND case, but /etc/dhcp/dhclient-enter-hooks.d/resolved undefines it (who's nasty now, eh?) even though resolved is masked.

The file existence check in the beginning of /etc/dhcp/dhclient-enter-hooks.d/resolved should be more thorough, i.e. it should ensure that resolved is enabled, rather than simply look for the existence of /lib/systemd/systemd-resolved. This works for me:

-if [ -x /lib/systemd/systemd-resolved ] ; then
+if [ -x /lib/systemd/systemd-resolved ] && systemctl -q is-enabled systemd-resolved ; then

Arguably, /etc/dhcp/dhclient-enter-hooks.d/resolvconf should implement a similar check, looking for /run/resolvconf/enable-updates as a condition for meddling with DNS settings. If desired, I'll file a separate bug for that package.

[GeekSmith (lixo-geeksmith)]

The hasty push to replace resolvconf with resolved is having disastrous consequences. User experience and system stability should be valued over forced migration to an unproven, poorly tested, and in many cases unwanted new system.

Name resolution is one of the fundamental building blocks of a networked system. I don't understand the eagerness to change it so quickly. It almost seems careless. One might argue that all of systemd has been pushed on the user in a like manner, but I won't go there...at least it's not upstart.

[Steve Langasek (vorlon)]

Nothing in your bug report explains why you have taken the initial step to disable systemd-resolved on your system. If it is simply that it is "unwanted", that is insufficient justification for us to support a configuration that involves disabling a component that the Ubuntu development team has selected for inclusion in the core Ubuntu experience.

The most likely resolution of this bug for 18.04 will be to remove the resolvconf package from the Ubuntu archive, and force its removal from systems on upgrade via Conflicts: from another relevant package (such as systemd).

If there are specific inadequacies in the behavior of resolved when it *is* enabled, please report these issues against the systemd package in launchpad, so that they can be resolved for the benefit of all users of Ubuntu 18.04.

[GeekSmith (lixo-geeksmith)]

Is the use of resolvconf and ifupdown without resolved an unsupported configuration in 17.10? Is resolved the only supported DNS configuration management system in Ubuntu 17.10? There are many users who value the control and flexibility of pre-resolved systems and do not want a local caching nameserver.

The inadequacy is that systemd breaks resolvconf. I highly doubt that resolvconf will be removed from the archives in 18.04 because resolved is clearly not ready. Look at the number of upgrade issues and bugs filed against it.

There are too many systems, mainly servers, that cannot or should not run a nameserver, not even a local one. resolved is not ready to service these systems.

Because resolved is packaged with systemd, one cannot choose to run systemd without resolvd. This coupling makes the system less flexible and more fragile. There's another inadequacy for you.

Not only have I discovered a bug, I have given the solution to the bug in two packages. My proposed solution does not affect performance nor hinder proper behavior in any way. It simplifies interface configuration. This simple fix should be favored over the removal of software that is stable, functional, and time-tested.

Denying the existence of the bug doesn't make it go away. I've reported the defect and provided a simple fix. The Ubuntu team can patch the scripts or leave them broken.

[Steve Langasek (vorlon)]

On Thu, Jan 25, 2018 at 11:02:35PM -0000, GeekSmith wrote:
> Is the use of resolvconf and ifupdown without resolved an unsupported
> configuration in 17.10?

resolvconf is in universe as of 17.10. In effect, yes, this is unsupported.

> Is resolved the only supported DNS configuration management system in
> Ubuntu 17.10?

Yes.

> There are many users who value the control and flexibility of
> pre-resolved systems and do not want a local caching nameserver.

Flexibility for flexibility's sake is not a goal of Ubuntu.

resolved is not configured as a caching nameserver; it is a stub resolver,
configured for the purpose of ensuring a stable DNS endpoint.

> The inadequacy is that systemd breaks resolvconf.

I'm sorry, but this is a circular argument which does not explain why you
are trying to use resolvconf in the first place.

> There are too many systems, mainly servers, that cannot or should not
> run a nameserver, not even a local one. resolved is not ready to service
> these systems.

What determines that a server "cannot or should not" run a local resolver?

Enabling a local resolver on servers in addition to desktops (where we have
already enabled dnsmasq for years) has been a common request.

If there are scenarios where it is not appropriate to run resolved, then we
should absolutely evaluate those and determine how they should be supported
in Ubuntu. However, they must be evaluated on their own merits, which means
that the technical details must be presented for consideration.

[GeekSmith (lixo-geeksmith)]

> Flexibility for flexibility's sake is not a goal of Ubuntu.

Hmmmm...flexibility for the sake of making a system work seems like a good goal. As a user of Linux for 18 years, and one that has used Ubuntu since its very first release, I can say with confidence that Linux is absolutely about choice and flexibility. Any distro that forgets this fact has limited or short-lived success.

> resolved is not configured as a caching nameserver; it is a stub resolver,
> configured for the purpose of ensuring a stable DNS endpoint.

Perhaps I've conflated resolved with the NetworkManager dnsmasq instance, which is most definitely configured as a caching resolver (and the first iteration of this type of default setup in Ubuntu, also a nightmare for crusty old *nix users). My understanding, based on the documentation, is that resolved acts as a caching resolver. systemd-resolved.server(8) states very clearly:

"systemd-resolved is a system service that provides network name resolution to local applications. It implements a caching and validating DNS/DNSSEC stub resolver, as well as an LLMNR resolver and responder."

> Enabling a local resolver on servers in addition to desktops (where we have
> already enabled dnsmasq for years) has been a common request.

Consider this a request to allow an Ubuntu system to work without dnsmasq or any other local caching resolver. I'm sure it's not the first nor the last.

There are definitely scenarios where running dsnmasq as a local (caching?) resolver is inappropriate and I'm happy to give a few scenarios, but that's not what _this_ _bug_ is about. This bug is about a defect in the resolvconf and resolved dhclient integration scripts.

Philosophical discussion aside, I have reported a valid bug that will definitely affect some number of Ubuntu users. I've also provided a safe, simple, straightforward fix to this bug. It's an improvement to the software.

Unless the fix is technically unsound, there is absolutely no reason not to implement it.

[Dimitri John Ledkov (xnox)]

May I backtrack a bit, and ask how is this system configured? And is it a desktop or a server configuration? Is it a fresh install or an upgrade?

Specifically, what I am confused about is how the networking is configured and where /etc/resolv.conf is pointing at.

I see that NetworkManager and ifupdown were mentioned. Do you have both of these configured and try to manage a) networking b) /etc/resolv.conf?

And what are the steps to reproduce this? Ideally with something simple and reproducible like a lxd container launched with $ lxd launch ubuntu-daily:artful test-container

Note that overriding make_resolv_conf() is a good thing, and it should not at all be used. None of our network configuration tools rely on make_resolv_conf() which is an extremely bad interface that writes out a static file and looses information, as it assumes that there is only ever one interface and one dhcp. Hence you will notice that everything (networkd+resolved, ifupdown+resolvconf, networkmanager) overridee make_resolv_conf.

I'm not quite sure what was done to your system, but resolvconf can be used to managed the /etc/resolv.conf and that is integrated with resolved. You will notice that resolved stub nameserver is pulled into resolvconf config, and thus resolvconf can own /etc/resolv.conf which is what happens on upgrades.

Please update the bug description on how to reproduce the problem you are seeing, without ranting or jumping to conclusions, or trying to make any unnecessary changes, because it is hard to understand what you are trying to achieve, which may have been made worse with unnecessary workarounds.

[GeekSmith (lixo-geeksmith)]

What I'm trying to achieve is this: use resolvconf instead of resolved in Ubuntu 17.10. There is a bug when one tries to do so.

This is a fresh install on a laptop using the desktop installation DVD.

resolv.conf points to /run/resolvconf/resolv.conf and is managed by resolvconf.

I have NetworkManager and resolvconf enabled for interfaces not managed by ifupdown, as shown in NetworkManager.conf in the initial report. I'm using ifupdown for ethernet interfaces, NetworkManager for Wi-Fi.

The steps to reproduce are:
1. Install and enable resolvconf
2. Stop and mask resolved
3. Configure NetworkManager for resolvconf and ifupdown
4. Configure an interface in ifupdown for dhcp
5. Connect to the network

I agree that overriding make_resolv_conf() is good and necessary. However, the scripts that do so currently do it blindly, even when the service they support is disabled. My patch adds a simple sanity check to ensure that disabled services don't run a script that will break DHCP for enabled services.

I've described what was done to my system. If you have further questions, please let me know. My goal is to use resolvconf in the place of resolved. In doing so, I uncovered a bug in the dhcp hook scripts for both resolved and resolvconf and I've provided a working patch for those bugs. The patches have been running successfully on my systems for months.

All I ask is for consideration of the patches I've provided, not a criticism of choosing resolvconf over resolved.

[GeekSmith (lixo-geeksmith)]

To distill this down to its most basic form, let's try this.

When resolvconf or resolved is installed but disabled, its DHCP hook script still executes and potentially interferes with the proper operation of other hook scripts.

My patch ensures that these hook scripts only run when the corresponding service is enabled.

[Launchpad Janitor (janitor)]

[Expired for resolvconf (Ubuntu) because there has been no activity for 60 days.]

[Frederic Van Espen (frederic-ve)]

I quite agree with @GeekSmith here. We disabled systemd-resolved on bionic and install ifupdown instead since we install this on server that have a static configuration and we don't want systemd-resolved to modify anything in the resolv.conf file.

Only during the initial installation time of the server we use dhclient to get an IP address. The result is that the resolv.conf file is not modified by dhclient despite systemd-resolved being disabled. To work around this issue, we simply deleted the /etc/dhcp/dhclient-enter-hooks.d/resolved file as we don't need it anyway.

A cleaner way of course it to actually test in the file if systemd-resolved is used, as @GeekSmith's patch does.

[Frederic Van Espen (frederic-ve)]

Just clarifying that we don't use the resolvconf package either, but just a plain static /etc/resolv.conf file.

[kalvdans (4-launchpad-kalvdans-no-ip-org)]

I agree with the change @GeekSmith suggests as well.

I'm running a system without network-manager, without netplan, without resolvconf. Only ifupdown and dhclient. I expect /etc/resolv.conf to be updated by the logic in /sbin/dhclient-script where it (correctly, in my system) assumes it is the only source of dns servers. I have reopened the bug.

[Steve Langasek (vorlon)]

> I'm running a system without network-manager, without netplan, without
> resolvconf. Only ifupdown and dhclient. I expect /etc/resolv.conf to be
> updated by the logic in /sbin/dhclient-script where it (correctly, in my
> system) assumes it is the only source of dns servers. I have reopened
> the bug.

I repeat myself:

  Flexibility for flexibility's sake is not a goal of Ubuntu.

That you have disabled a base component of Ubuntu and the resulting system does not function as you expect is not a valid bug report against Ubuntu.

  If there are scenarios where it is not appropriate to run resolved, then we
  should absolutely evaluate those and determine how they should be supported
  in Ubuntu. However, they must be evaluated on their own merits, which means
  that the technical details must be presented for consideration.

[GeekSmith (lixo-geeksmith)]

kalvdans -- they don't get it. Inflexibility for inflexibility's sake is the new goal. My proposed fix is very simple and safe, but because it allows flexibility it's not going to be accepted.

Feel free to edit the broken script in the package to enable your setup to work properly. I've done this on several systems I administer and they are running as one would expect now.

[Thayne (thayne-u)]

> If there are scenarios where it is not appropriate to run resolved, then we
  should absolutely evaluate those and determine how they should be supported
  in Ubuntu.

Maybe there is a better way to do this, but here is my situation. I need to be able to send DNS queries to multiple dns servers in parallel and use the first response I get back (so that I use the fastest/closest dns server). On 16.04 I did this by running dnsmasq with the "all-servers" option enabled. afaict, resolved doesn't have an equivalent option. The nameservers come from a supersede rule in the dhclient configuration. But on 18.04 if resolved is running, then dnsmasq uses resolved as the nameserver, which isn't what I want, but if resolved isn't running then I run into this bug and dnsmasq doesn't get any nameservers.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Steve Langasek (vorlon)]

Hello,

On Tue, Nov 26, 2019 at 04:42:44PM -0000, Thayne wrote:
> Maybe there is a better way to do this, but here is my situation. I need
> to be able to send DNS queries to multiple dns servers in parallel and
> use the first response I get back (so that I use the fastest/closest dns
> server).

That's a pretty unusual requirement, which at the very least means that more
of your bandwidth is being used for DNS traffic than would otherwise be the
case. Can you help me understand the motivation underlying the requirement?
Do you have DNS servers in your configuration which are always farther /
slower? If so, why do you have them in your configuration at all? Do you
have a particular application requirement for low-latency DNS resolution -
and if so, wouldn't the use of a caching local resolver (a configuration
which resolved supports, and which we enable by default) have more of an
impact on satisfying that requirement?

We should certainly ensure that your use case is addressed with resolved
itself, and without the need to run dnsmasq as an alternative local
resolver.

[Thayne (thayne-u)]

> Can you help me understand the motivation underlying the requirement?

The motivation is that if the current dns server goes down, we don't have to wait for the request to timeout and make another request to the next server. Before resolved this was more of a problem since libc would always try to use the first server if it was down and didn't have any memory, so we would add the timeout for the dns lookup to each dns query. But I understand that resolved does remember if a server is down and continues to use that one, but afaict it never switches back, and that is a problem, as I'll explain below.

> Do you have DNS servers in your configuration which are always farther /
slower? If so, why do you have them in your configuration at all?

Yes. We have one DNS server in each availability zone (AWS). We would prefer to use the DNS server in the same availability zone as the host, but want to fall back to one of the others if that one becomes unavailable.

> Do you have a particular application requirement for low-latency DNS resolution - and if so, wouldn't the use of a caching local resolver (a configuration which resolved supports, and which we enable by default) have more of an impact on satisfying that requirement?

The less time it takes for our application servers to receive an answer the better. Bandwidth for DNS traffic is very cheap and the overhead for DNS CPU is also very inexpensive. We want to have the ability to use the fastest answer from an upstream DNS server as possible. We are also want redundancy so that we can remain unimpacted by planned and unplanned maintenance / issues. A local DNS cache helps with the fast response, but it does not help with an upstream servers availability.

[Bast1aan (bast1aan)]

I've spent a lot of time discovering this issue, and now it is bugging me every time there is an update of systemd, it wants to reinstall the file again.

Actually this should be solved by making systemd-resolved a separate package so it can actually be UNINSTALLED including files that disturb other packages, instead of disabled...

[Bast1aan (bast1aan)]

If you google a bit around then you will find many issues with systemd-resolved that could be a reason you want to be able to not use it. So I have a strong opinion that Ubuntu should keep the flexibility to be able to use alternative resolving configurations.

[Thayne (thayne-u)]

In this systemd github issue https://github.com/systemd/systemd/issues/5755 in particular is problematic darkstar states "the DNS servers are supposed to be exactly equivalent". But as far as I can tell systemd doesn't have any provision for temporarily falling back to a different nameserver if the main one is unavailable and reverting back when it becomes available again. Besides the scenario I listed in my previous comments, this also prevents using a local nameserver as the main nameserver but falling back to an external nameserver if the local one becomes unavailable, with recovery. In that case the two nameservers may not even return the same responses, but having a working nameserver that returns public records is better than no nameserver at all. Again, if there is a way to accomplish failovers with recovery with resolved, I'd love to know how to do it.

Maybe systemd-resolved will get functionality that supports these use cases, but right now ifaict it doesn't. And even when it does, will it get backported to LTS versions of ubuntu such as 18.04?