systemd-resolve breaks resolution of local network hostnames

[Forest]

After upgrading to Ubuntu 17.04 (zesty), resolution of my local network's host names is completely broken. Apparently the upgrade replaced my existing resolver with systemd-resolve, which deliberately refuses to pass "single-label" domain names to my domain name server. That is the server where all my network's host names are kept, so I can no longer resolve any of them.

Apparently, this is yet another example of Poettering's upstream decisions causing denial of service to people who have been saddled with his malware.
https://github.com/systemd/systemd/issues/2514#issuecomment-179203186

Would someone sensible please put a stop to this forced breakage during upgrade, and advise on how to fix it now that the damage has been done?

[Dimitri John Ledkov (xnox)]

you should be able to opt-into UseDomains by creating

/etc/systemd/network/usedomains.network

[DNS]
UseDomains=true

I believe. does above help you resolve things?

Can you tar up and attach the tarball of /run/systemd to investigate which links you have available, and what domains they have?

[Forest (foresto)]

According to the systemd documentation, UseDomains only affects systems that get their network setup from DHCP, which is not the case at my site. Furthermore, it is documented as a way to add a DNS search domain, which would not help resolve hosts that have no domains.

So far, the only way I have found to fix the breakage is to manually disable systemd-resolve and configure a resolver that actually works properly.

[Dimitri John Ledkov (xnox)]

You can adjust /etc/systemd/resolve.conf to add Domains= that you want to use for single domain resolution for non-DHCP specified DNS server.

[Dimitri John Ledkov (xnox)]

Sorry, typpo /etc/systemd/resolved.conf

https://www.freedesktop.org/software/systemd/man/resolved.conf.html#Domains=

[Forest (foresto)]

No, that won't help either.

You don't seem to understand: Adding a domain search list will not help because the local machines do not have domains. They only have names, like "host1" or "beetle", even on the network's DNS servers.

Since systemd's resolver refuses to consult the name servers for names like these, it completely breaks name resolution on networks like this.

[Dimitri John Ledkov (xnox)]

Ok, i see what you mean now. I will test this locally to verify.

[Marc MAURICE (mmaurice)]

Same here. After upgrading somes pcs in my company from 16.04 to 17.10 : lots of local services are not working anymore.

[Dimitri John Ledkov (xnox)]

There is a regression with dns handling between xenial and zesty. There are updates/improvements made in artful that will be published soon. It may not solve your issue, but maybe/hopefully it will improve things a lot.

systemd (234-1ubuntu2) artful; urgency=medium

  * Set UseDomains to true, by default, on Ubuntu.
    On Ubuntu, fallback DNS servers are disabled, therefore we do not leak queries
    to a preset 3rd party by default. In resolved, dnssec is also disabled by
    default, as too much of the internet is broken and using Ubuntu users to debug
    the internet is not very productive - most of the time the end-user cannot fix
    or know how to notify the site owners about the dnssec mistakes. Inherintally
    the DHCP acquired DNS servers are therefore trusted, and are free to spoof
    records. Not trusting DNS search domains, in such scenario, provides limited
    security or privacy benefits. From user point of view, this also appears to be
    a regression from previous Ubuntu releases which do trust DHCP acquired search
    domains by default.
    Therefore we are enabling UseDomains by default on Ubuntu.
    Users may override this setting in the .network files by specifying
    [DHCP|IPv6AcceptRA] UseDomains=no|route options.
  * resolved: create private stub resolve file for integration with resolvconf.
    The stub-resolve.conf file points at resolved stub resolver, but also lists the
    available search domains. This is required to correctly resolve domains without
    using resolve nss module.
  * Enable systemd-resolved by default
  * Create /etc/resolv.conf at postinst, pointing at the stub resolver.
    The stub resolver file is dynamically managed by systemd-resolved. It points at
    the stub resolver as the nameserver, however it also dynamically updates the
    search stanza, thus non-nss dns tools work correctly with unqualified names and
    correctly use the DHCP acquired search domains.
  * libnss-resolve: do not disable and stop systemd-resolved
    resolved is always used by default on ubuntu via stub resolver, therefore it
    should continue to operate without libnss-resolve module installed.

 -- Dimitri John Ledkov <email address hidden> Fri, 21 Jul 2017 17:07:17 +0100

[Dawid Wróbel (dawidw)]

This is still a problem in 18.04. Like the OP, I have localhosts on my LAN, but have no local domain configured. I had to disable systemd-resolved on each of my LXC containers. What's worse, resolvconf creates an empty /etc/resolv.conf file. So I had to create resolv.conf by hand and point to DNS. Again, on each of my containers.

This is a major regression. If the systemd-resolved is to be kept, at least have some decency to explicitly note in 18.04 LTS release notes that local hostname resolution requires local domain to be configured for name resolution to work.

[Evgeni (ev-mp)]

One more affected usecase - wired LAN connection is not working properly for VMWare Virtual Machine.
I use laptop with Win10 and run Ubuntu 18.04 as guest OS with VMWare.
Everything worked fine with Ubuntu 16 and 14, but with the current version the DNS resolution is now broken and I cannot run even "sudo apt update"

Can someone publish instructions how to work around this issue ?

Evgeni Raikhel <email address hidden>

P.S
Interestingly enough, FireFox is able to reach internet, probably by circumventing the built-in DNS resolution mechanism.

[Pedro Côrte-Real (pedrocr)]

Just got bitten by this bug on 18.04 upgrade. This is the default config on LEDE/OpenWRT that gets broken by this upstream decision. Ubuntu should definitely ship a fix of some sort.

[Pedro Côrte-Real (pedrocr)]

After testing further I get somewhat random results. Sometimes it fails and sometimes after running "sudo service systemd-resolved restart" I can resolve domains. Very weird.

[Tom Fields (udzelem)]

Hi,

Ubuntu 18.10 using systemd version 239-7ubuntu10.5:

I also see random results for dot-less domain or host names, i.e. it seems after a certain timeout, a hostname is not found e.g. when trying to establish an SSH connection to that machine using a single-label name. Immediately followed by a second try however, all is working again.

This looks like a time-out or caching issue.

Quoting https://github.com/systemd/systemd/issues/6224#issuecomment-312639959:
"LLMNR lookups for missing names need to time out (because it is a true multicasting peer-to-peer protocol, and a name that doesn't exist means that there's nobody responding to an LLMNR query), but the host command on the client side times out even earlier than that, and then doesn't continue with the search domains, as search domains have to be honoured client-side, i.e. inside the "host" command."

I have now disabled LLMNR in /etc/systemd/resolved.conf and will observer in the next days if this changes the behaviour.

[Tom Fields (udzelem)]

IIRC what could be happening is that the additional timeout introduced by systemd-resolved first doing the newly implemented LLMNR lookup (which fails if not implemented by any device) causes some network tools to not even try doing a DNS lookup using the DHCP- or user-supplied DNS search domain suffixes.

The second invocation then maybe finds a cached result which is however invalidated after some time - thus the varying or seemingly random results.

[Dmitrii Shcherbakov (dmitriis)]

Try using this:

https://www.freedesktop.org/software/systemd/man/resolved.conf.html#Domains=
"Use the construct "~." (which is composed of "~" to indicate a routing domain and "." to indicate the DNS root domain that is the implied suffix of all DNS domains) to use the system DNS server defined with DNS= preferably for all domains."

cat /etc/systemd/resolved.conf.d/99-local-fqdn-wa.conf
[Resolve]
DNS=192.0.2.1 192.0.2.2 192.0.2.3
Domains=~.

[Martin Pecka (peci1)]

I successfully work around this issue by the steps described at https://askubuntu.com/a/907249/153828 :

Disable and stop the systemd-resolved service:

sudo systemctl disable systemd-resolved.service
sudo systemctl stop systemd-resolved

Then put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf:

dns=default

Delete the symlink /etc/resolv.conf

rm /etc/resolv.conf

Restart network-manager

sudo service network-manager restart

[ironstorm (ironstorm-gmail)]

Thanks Martin, your solution to disable systemd-resolve allowed me to finally resolve hosts on my local network again... I was using a temporary workaround of replacing "echo 'nameserver 192.168.1.1' > /etc/resolv.conf", however that doesn't survive reboots.

This issue should be addressed by Ubuntu devs, systemd is pretty seriously f-ing up a basic form of usability in being able to resolve machines (and devices) by name on a local home network.

[Adrianvg (adrianvg)]

Plain vanilla install of Kubuntu 20.04 and this still seems to be a problem.
Tried @peci1 solution, but didn't do anything. I still can't resolve names to IP-addresses.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue