systemd-resolved appends 127.0.0.53 to resolv.conf alongside existing entries

[Anders Kaseorg]

systemd-resolved, or more precisely the hook script /lib/systemd/system/systemd-resolved.service.d/resolvconf.conf, causes resolvconf to add 127.0.0.53 to the set of nameservers in /etc/resolv.conf alongside the other nameservers. That makes no sense because systemd-resolved sets up 127.0.0.53 as a proxy for those other nameservers. The effect is similar to bug 1624071 but for applications doing their own DNS lookups. It breaks any DNSSEC validation that systemd-resolved tries to do; applications will failover to the other nameservers, bypassing validation failures. And it makes failing queries take twice as long.

/etc/resolv.conf should have only 127.0.0.53 when systemd-resolved is active.

[Martin Pitt (pitti)]

The primary purpose of adding 127.0.0.53 to resolv.conf is for client software that wants to do DNS resolution by itself instead of using NSS -- most notable example is Google Chrome, and third-party software which is statically linked (e. g. Go).

However, other software like NetworkManager or isc-dhcp also calls resolvconf and adds name servers picked up by them -- as they don't talk to resolved directly, resolved reads their DNS servers *from* resolv.conf.

But, software which does its own DNS lookups like the above have to do their own DNSSEC validation too -- you can't both chose to *not* use NSS *and* rely on NSS to do DNSSEC for you..

So, this is indeed a wart, but not easily fixed, and also not that important IMHO. Not using NSS is already broken to some degree, as you also ignore things like nss-{winbind,docker,ldap} etc.

[Anders Kaseorg (andersk)]

DNS resolution outside NSS shouldn’t be dismissed as an edge case for software that’s too conceited to use NSS. NSS only exposes A, AAAA, and PTR records. There’s plenty of software in the official archive that needs other records from DNS (off the top of my head: SRV, TXT, MX, SSHFP, AFSDB) and cannot get them from NSS.

> resolved reads their DNS servers *from* resolv.conf.

Right, so perhaps when resolvconf is in use, systemd-resolved should read those from /run/resolvconf/resolv.conf directly, and /etc/resolv.conf should be a symlink to /lib/systemd/resolv.conf rather than /run/resolvconf/resolv.conf?

> you can't both chose to *not* use NSS *and* rely on NSS to do DNSSEC for you.

Why not? It was working with NetworkManager managing a dnsmasq, since NetworkManager installed the local proxy as the only nameserver visible in resolv.conf, and it would work again if systemd-resolved did the same.

This will also be needed to fix problems like https://github.com/systemd/systemd/issues/3421 for programs that cannot use NSS.

[Willem Toorop (willem-l)]

Unfortunately the DNS interface of current systemd-resolved strips DNSSEC, so applications that do DANE validation still have to target the upstreams directly.
I have filed a bug about this: https://github.com/systemd/systemd/issues/4621

[Martin Pitt (pitti)]

As explained, 127.0.0.53 must be in /etc/resolv.conf in order to support Chromium and other software which does not NSS -- that is the whole raison d'être for providing a local stub server.

In Yakkety with NetworkManager only the 127.0.1.1 dnsmasq server is in /etc/resolv.conf, so that isn't affected. In Zesty NM now does not manager resolv.conf itself any more but just sends the acquired name servers to resolved.

Thus, is there anything left from the original bug report that still needs to be addressed?

[Willem Toorop (willem-l)]

Au contraire,

I argue to not fix this issue and keep the other upstream resolvers alongside 127.0.0.53 in /etc/resolv.conf, because 127.0.0.53 will break applications that need to do DNSSEC validation themselves (for example for DANE).

Once systemd-resolved has been fixed to provide the DNSSEC data alongside the answer, I agree with Anders that the other resolvers should go.

[Anders Kaseorg (andersk)]

Martin: I still wonder what the point of having resolvconf is, if it’s only ever supposed to be used to manage 127.0.0.53, and every other use of resolvconf will lead to this bug resurfacing. I still propose that systemd-resolved should read from /run/resolvconf/resolv.conf without adding 127.0.0.53 to it, and /etc/resolv.conf should be a symlink to /lib/systemd/resolv.conf rather than /run/resolvconf/resolv.conf.

Willem: If systemd-resolved isn’t suitable to be the only nameserver in resolv.conf, then it isn’t suitable to be in resolv.conf at all. I would rather either see these bugs worked out during the zesty cycle, or have systemd-resolved removed entirely, than leave the system in a half-broken state where the bugs in systemd-resolved get (probabilistically?) masked by the presence of other nameservers in resolv.conf.

In its current state, systemd-resolved is thoroughly broken on account of (at least) bug 1647031. I had to downgrade network-manager because switching to a resolver that doesn’t follow CNAME records breaks way too much of the internet. This might not have been noticed with other nameservers in resolv.conf. (Although now that it has, I’m getting increasingly concerned by the switch not having been reverted yet…)

[Doug Goldstein (cardoe)]

So this issue bit me in a weird way. I had my bridge called "xenbr0" and the result of these two interacting gives me absolutely no DNS resolution since the /etc/resolv.conf just pointed to 127.0.0.53 and didn't include the DNS server from my "xenbr0". Renaming the bridge to "renbr0" caused it to work. Why "renbr0"? Because in /run/resolvconf/interface/ there is a file called systemd-resolved that I was guessing was stopping combining the files once it hit the systemd-resolved.

[Vincent Fortier (th0ma7)]

From the man page systemd-resolve can run in 3 mode of operations.

1) Ubuntu 17.10 default - The default is to list the 127.0.0.53 DNS stub (see above) as only DNS server. This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to systemd-resolved. This mode of operation is recommended.

2) WHAT YOU WANT: systemd-resolved maintains the /run/systemd/resolve/resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf and is always kept up-to-date, containing information about all known DNS servers....

3) PRE-17.04: Alternatively, /etc/resolv.conf may be managed by other packages, in which case systemd-resolved will read it for DNS configuration data. In this mode of operation systemd-resolved is consumer rather than provider of this configuration file.

The fix:
root@localhost:~# ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 29 mar 7 20:20 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
root@localhost:~# rm -f /etc/resolv.conf
root@localhost:~# ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
root@localhost:~# ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 32 mar 8 07:30 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

Finally firefox started working properly along with all my command line tools...

[Leo Raczka (keseszel)]

Hello everyone. Sorry for my english.
I have trouble with this f.. DNS, because 127.et around.
I change this for 8.8.8.8 - then FF working properly. I shut down my Ubuntu and DNS znowu 127.etc when I running Ubuntu. This sick.

[Mike Loebl (mloebl)]

I appreciate Vincent's post. I upgraded to 17.04 and DNS resolution has been absolutely terrible; slow, failed lookups, etc. Finally realized it was using 127.0.0.53 as the resolver. I did the method 2 above and DNS back to normal.

[Tom Kidman (tom-kidman)]

I had this problem too. I've upgraded 2 machines from kubuntu 16.10 to 17.04 and afterwards both were unable to resolve DNS. The only line in /etc/resolv.conf was 'nameserver 127.0.0.53'. As Mike wrote, following Vincent's fix above fixed the issue for me. Thanks Vincent :)

[Fred Senior (freddyp)]

Other *nix machines and Macs their resolve.conf on my LAN just append the dns server address that is configured om their network manager or network preferences. Why Ubuntu does not follow this I do not understand.

It stops my Ubuntu devices resolving FQDN's on the the LAN. I have to manually edit /etc/resolv.conf to fix this.

[Alroger Filho (alroger-cafe-ti)]

Same problem here. Editing resolv.conf or #8 post solves it.

I almost lost a RAID today because postfix was not able to send mdadm notifications -
delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp.gmail.com type=MX: Host not found

I use /etc/network/interfaces, nothing in networkmanager.
This is serious. Ubuntu shouldn't just start changing things like this, specially if it breaks a server.

Using a clean 17.04 install here.

[Alexander List (alexlist)]

Using a clean 17.04 install, I could observe the same problem.

Against convention, our company is using .local as the internal DNS TLD.

As mentioned in #8 and #13 above, /etc/resolv.conf was symlinked to

root@dell-e5470:~# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Apr 5 09:19 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

which contains

---
# cat /run/resolvconf/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
---

After removing the symlink /etc/resolv.conf and linking like this:

# ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

...thinks started working again. Now the file contains the DNS received via DHCP:

-----
# cat /etc/resolv.conf
# This file is managed by systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known DNS servers.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 1.2.3.4
search local.example.com example.local
-----

[Dimitri John Ledkov (xnox)]

@alexlist

At least for me, I do get

nameserver 127.0.0.53
search local.example.com example.local

In my resolv.conf by default. As in the exppected domains are listed in the search search stanza in the ../run/resolvconf/resolv.conf for my company network. As well as /run/systemd/resolv/resolv.conf.

Could you please check the following:
* move existing /etc/resolv.conf out of the way
* specify:
nameserver 127.0.0.53
search local.example.com example.local

And check that things work?

Also please do not test just the 17.04 clean install, but also upgrade systemd to the update shipped in zesty-updates first.

[Dimitri John Ledkov (xnox)]

If search stanzas are not generated for the 127.0.0.53 in resolv.conf, we need to open a new bug report I believe.

[Otto (otto11235)]

In my case after a 17.04 installation I needed a HOSTS file in order to resolve local hosts. After applying the fix in post #8 both local and remote hosts were resolved by the router DNS server (as was the case in the past).

[Otto (otto11235)]

Update to post #17...
The fix unfortunately did not fix all DNS problems. nslookup and dig can resolve local hosts, BUT ping and ssh both report 'Name or Service not known'. Looks like the HOSTS is still required.

[Antoine Bertin (diaoulael)]

I've been struggling with this issue... It seems I have to append the localdomain to my local names for the resolution to work :

nslookup somename : SERVFAIL, 127.0.0.53 didn't proxy to my router
nslookup somename.localdomain : OK, answer from my router

Why somename request isn't forwarded to my router?

Link 2 (eth0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.200
          DNS Domain: localdomain

I'm ok for some proxying to take place if that proves useful but so far it's only been a PITA for me so I disabled it following Vincent's answer.

[Kevin Rudloff (krudloff)]

Hello everyone! I had the same issue. Google Chrome and Firefox just loaded Google's websites.
I went to resolv.conf to see what was happening and I saw a new configuration. So I decided to add Google's DNS and it worked again but just till I restarted my laptop.
I don't know much of programming but the solution purposed by Vincent Fortier (th0ma7) worked for me. Thanks!

[Bernhard (xro)]

Hi,

I just upgraded from Ubuntu 16.10 to 17.04 and observed the following systemd-resolve behaviour that might be related to this bug:

--> Any domain listed after the "search" keyword in /etc/resolv.conf stops being resolved by systemd-resolve.

respectively

--> Any domain listed as "DNS Domain:" when querying "systemd-resolve --status" stops being resolved by systemd-resolve

I confirmed with tcpdump that systemd-resolve will not query my local or any dnsserver for any subdomains for any domain listed under the "search" keyword.

Changing the domains after the search keyword in /erc/resolv.conf immediately changes which domains systemd-resolve will ignore and not resolve.

The search domain is normally automatically set in response to receiving an "option domain-name" from the local dhcp server and normally used to translate requests for www to e.g. www.localdomain.at if localdomain.at was the domain-name sent by the DHCP server.

[Rick Timmis (rick-timmis)]

Here is my use case experience, after running a clean install of Kubuntu 17.10.

Initially everything was perfect, and for at least 10 days my machine booted perfectly fine and DNS working without issue.

However, after installing virt manage Virtual Machine Manager, like so

sudo apt-get install qemu-system qemu-kvm virt-manager wireshark

I configured some additional Virbr networks ( see below ) The following day after a reboot, I had no DNS resolution, I hacked /etc/resolv.conf to add a nameserver. All good until reboot, and then back to square one.

Once I discovered this bug things became clearer, and I have applied fix at post #8 above

I figure that my use of virt-manager might be an interesting edge case that break things with systemd-resolve. Below is the output from status, which highlights the issue, there are no DNS servers listed.

Hope this info is useful

ricktimmis@ricktimmis-Latitude-E6430:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 23 (vnet1)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 17 (vnet0)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 15 (vnet5)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 14 (vnet4)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 13 (vnet3)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 9 (virbr0-nic)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 8 (virbr0)
      Current Scopes: LLMNR/IPv4
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 7 (virbr1-nic)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 6 (virbr1)
      Current Scopes...

[Rick Timmis (rick-timmis)]

Today I discovered some additional information in regard to this issue.

I am using Virt-Manager and KVM, I have a Br0 bridge configured to start onboot. When this is the case name resolution as above fails.

If I delete the bridge using VMM name resolution starts working ( after a reboot ). Put the bridge back in and it fails.

I do have 3 network interfaces, 1 wlan, 1 ethernet, and and additional usb ethernet configured as eth1.

I've configured the 3 interfaces using a persistent-rules file in /etc/udev/rules.d/ so that they bind to eth0, wlan0 and eth1.

By configuring my VMM Br0 interface to Hotplug, so that it does not start onboot, this has resolved the issue. However, on Ubuntu 16.04 everything worked as expected including having the Br0 bridge start onboot.

Can't say I understand what magic SystemD witchcraft is going up here, but I hope the above info provides some useful clues.

[WhyteHorse (whytehorse)]

This bug just compromised every ubuntu machine on my network. It falsely says that DNSSEC is not supported by the nameserver and resorts to non-DNSSEC resolution. So every machine on my network just accepted bogus DNS replies from a MITM. Thanks.

[damianreloaded (damianreloaded)]

I had a dns server configured in my thinkpad and 127.0.0.1 set in resolv.conf. After upgrading from 16.04 to 17.04 resolv.conf gets reset to 127.0.0.53 which in my humble opinion and extensive experience makes absolutely no sense whatsoever. At the very least this file should be left alone.

[Kyoku (kyoku)]

I set up an Ubuntu 18.04 test server today as an LXD host machine and ran into this issue.

network:
  version: 2
  renderer: networkd
  ethernets:
    ens33:
      dhcp4: no
      dhcp6: no

  bridges:
    br0:
      interfaces: [ens33]
      dhcp4: no
      dhcp6: no
      addresses: [192.168.1.101/24]
      gateway4: 192.168.1.1
      nameservers:
        addresses: [192.168.1.1, 8.8.8.8, 8.8.4.4]

root@bionic:/home/bionic# systemd-resolve --status
Global
         DNS Servers: 192.168.1.1
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 6 (tun0)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 5 (veth0OI7SQ)
      Current Scopes: LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (br0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
 DNSSEC supported: no
         DNS Servers: 192.168.1.1
                      8.8.8.8
                      8.8.4.4

Link 2 (ens33)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

root@bionic:/home/bionic# ping google.com
ping: google.com: Temporary failure in name resolution

Also the error message is misleading, there's nothing "temporary" about this failure. It's permanent.

I can reach IP addresses directly but all DNS lookups are failing, preventing me from doing upgrades and normal operations.

[pierre (dubiaj)]

FWIW, you can also disable systemd-resolve altogether since it's working so poorly. The workaround for #8 works, but ping and ssh are still broken with it which is far from ideal.

I found this: https://askubuntu.com/a/907249 which seems to break vpn dns resolution so beware.

[Steve Langasek (vorlon)]

Anders, are you still seeing this issue with currently-supported versions of Ubuntu? I have argued elsewhere that the correct thing to do for 18.04 is to forcibly remove resolvconf on upgrades, switching /etc/resolv.conf to exclusively point to /run/systemd/resolve/stub-resolv.conf. In the common case, however, this should not make a difference. What other sources of DNS settings do you have that are feeding directly to resolvconf instead of resolved in 16.10 and later?

[Anders Kaseorg (andersk)]

On my bionic system, I had to manually remove resolvconf on account of bug 1713457. I think it was not being autoremoved because other packages like isc-dhcp-client, pppconfig, vpnc-scripts have Suggests: resolvconf.

So I don’t think I’m seeing this particular issue. However, the lack of resolvconf cooperation means that openconnect run from the command line fights with systemd-resolved over /etc/resolv.conf. I’m sure that all kinds of nonsense bugs like that are waiting to be rediscovered.

For the record, I’m still really quite angry at systemd-resolved for a number of issues that upstream seems to have no interest in addressing. Its own DNSSEC validation is disabled in Ubuntu because it’s broken (bug 1628778), yet it also breaks DNSSEC validation in applications downstream of it (https://github.com/systemd/systemd/issues/4621). It breaks ‘dig +trace’ (https://github.com/systemd/systemd/issues/5897). It doesn’t work with resolvconf in a sane way. The many previous issues I’ve had with it took way too long to be addressed. And it does nothing useful that dnsmasq didn’t do! (I even had NetworkManager’s dnsmasq configured to to DNSSEC validation and it worked just fine.) I’m not one of those anti-systemd people who hates change for the sake of hating change. But systemd-resolved is an actual flaming garbage pile that needs to be vitrified and launched into the sun.

[henry eko (hamster97)]

Is this behavior related to the bug?
I have running router on 192.168.31.1

Then, I run a test setup of bionic on virtual box as a squid proxy.
Everything fine, almost.

I found every local network names are failed to be resolved by the new bionic installation, but outside local network names are resolved.
It is as if the system resolve the domain name to another dns server beyond configuration.

root@rld-proxy:/etc/NetworkManager/dispatcher.d# systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (enp0s3)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.31.1

root@rld-proxy:/etc/NetworkManager/dispatcher.d# nslookup dm97-pandorabox
Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find dm97-pandorabox: SERVFAIL

root@rld-proxy:/etc/NetworkManager/dispatcher.d# nslookup dm97-pandorabox 192.168.31.1
Server: 192.168.31.1
Address: 192.168.31.1#53

Name: dm97-pandorabox
Address: 192.168.31.1
Name: dm97-pandorabox
Address: fd76:bf99:23d7::1

Any thoughts?
If any work around produce another bugs,
I guess it is safe for me to pending upgrade to bionic.

[fedorowp (fedorowp)]

Given the number of bug reports, and that this causes delayed breakage in server environments, perhaps this bug priority should be raised from "Low" to "Medium"?

I encountered this delayed failure in an LXC container, following upgrading the server and LXC container from Ubuntu 17.10 to Ubuntu 18.04.

Workaround
----------
Leandro Noskoski on askubuntu.com provided this work-round.

Edit /etc/systemd/resolved.conf, adding your desired DNS server:

change this:

[Resolve]
#DNS=

to this (but use the one you want - this is an example):

[Resolve]
DNS=192.168.1.152

after that, restart the service:

service systemd-resolved restart

DNS resolving should -NOT- be broken for containers by a single version OS upgrade
----------------------------------------------------------------------------------

Unfortunately this is a result of systemd violating 3 of 9 tenets of the Unix Philosophy:

1 - Small is beautiful.

2 - Make each program do one thing well.

6 - Use software leverage to your advantage.

[Jeff Carr (basilarchia)]

I vote for changing it from "Low" to "Insanely broken and wrong".

Even the workaround to set DNS in resolved.conf is still wrong. There is a reason that every dhcp server on earth is configured to provide a DNS server for a reason.

Jeff Carr
CEO WIT.COM Inc.
Formerly co-founder Digital Ocean

[Falk (andreas-mockel)]

I could not agree more with Jeff Carr.
This is a really important function.

From my pov this is not a low priority bug.

--
Regards Falk

[Dell (dell-humbert)]

The first 18.04 computer added and it can not communicate with any of the other servers on the local network because it does not honor the dns entry from dhcp. How this bug is not already set to "Insanely broken and wrong" escapes me.

[Marten (martenp-bewildered)]

This is a blocking issue for non-technical users (even for technical users, it is a massive PITA). It must be addressed as a matter of priority.

[pavera (pavera-b)]

I'll add my name to the list of people saying this is a blocker. I am currently working on a massive upgrade of 1000+ systems to Ubuntu 18.04 planned for early next year. Local dns resolution is an absolute must, and it must work out of the box. I can't update /etc/resolv.conf symlinks on thousands of systems manually. Even in an automated way, this is cumbersome and prone to break unexpectedly when updates get applied and revert the manual change. My DHCP server hands out a DNS server, and I expect my clients to ask that DNS server for all name resolution. How this long held tenant of network operation is being so horribly violated by systemd-resolved I don't understand. Did no one test this once? This is "Insanely broken and wrong".

Please fix this bug ASAP.

[Alessandro (moskante)]

I also think it is a blocker. I am a non technical user, I have lost one working day for this bug. Dns resolution should work out of the box! I solved following the instruction in this post - not even sure it is the best fix, for now it seems to work.

https://answers.launchpad.net/ubuntu/+question/665374

[Chris Gutierrez (gcgutier)]

I'll chime in as well. This bug is absolutely ridiculous. I lost half a day to this.

I completely agree with Marten in 35:
> This is a blocking issue for non-technical users

I thought I had some nasty bug with my alexa to ngrok to django set up. Turns out, name lookups were just slow, causing my Alexa skill to time out. Once I figured out the cause, it was a 10m fix, but the hours of troubleshooting python packages to figure out the issue was just a waste.

Thank you to those of you who provided the details for the fix. Saved me a bunch of time.

To anyone that stumbles on this, I followed 8, 14 and 31 above, removing the file, relinking, and editing the systemd file to add a nameserver.

Here were my steps, YMMV:
sudo rm -f /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
sudo nano /etc/systemd/resolved.conf

then added the google nameservers to my file, adding this line.
nameserver 8.8.8.8

Good luck!

[clancey (786464699-z)]

Now, I wirted a job to listen the servername, if it shows 127.0.0.53, the job will change it to I want...

[klogg (joculator-gmail)]

Same issue here with 18.10
The bug is here and ignoring it for 2 years is just ridiculous

[tack22 (tack22)]

Another vote here to increase the importance from low to "Insanely broken and wrong". Let's get this fixed.

[Mike L (mikerl)]

Broke my Xfinity login in Thunderbird and Evolution. Needs some attention. Running Ubuntu 18.10.

[Mike L (mikerl)]

See https://askubuntu.com/questions/1098414/18-04-unable-to-connect-to-server-due-to-temporary-failure-in-name-resolution/1099311 for workaround.

[Sonny (aadityabhatia)]

openssh relies on RRSIG records to verify the remote key using DNSSEC and SSHFP resource records. See VerifyHostKeyDNS under ssh_config. systemd-resolve breaks this.

Here is a detailed blog article that covers the issue in depth:

https://moss.sh/name-resolution-issue-systemd-resolved/

[Drew (1121redblackgo)]

February 4th, 2019: Lost a day to this bug

[Adrian Wilkins (adrian-wilkins)]

My problem with systemd-resolved :

* My router hands out a nameserver (itself) via DNS
* When I'm inside my network, I want my router to resolve IP addresses for my domain
* When I'm outside my network, I want the public DNS to resolve them

First lookup works fine! Then systemd-resolved (I guess) caches the secondary lookup it's doing via public DNS servers and returns the public IP address for the server. This is not what is desired, and breaks a bunch of stuff (like SSH to my servers using names from inside my network).

Never had this problem with dnsmasq, because it apparently respected my network config and didn't do an end-run around the DNS servers it was handed by DHCP.

[David Litster (mrlitsta)]

Also affects me.

I cannot believe that this is still set to low. This is yet another case of the systemd circus-tools causing unexpected behavior for long-time linux users.

This breaks local DNS resolution in my homelab. I should NOT have to do a Google search to fix this, it should just work properly (like it used to).

Fresh install of 18.04.2. Notice that systemd is incapable of giving me a lookup on my local LAN:

#CORRECT WHEN SPECIFIED MANUALLY
david@server:~$ nslookup ansible-nas 10.0.1.1
Server: 10.0.1.1
Address: 10.0.1.1#53

Name: ansible-nas
Address: 10.0.1.37

#SYSTEMD-RESOLVD PROXY IS BROKEN
david@server:~$ nslookup ansible-nas
Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find ansible-nas: SERVFAIL

#AND YET IT CLAIMS IT'S USING MY ROUTERS DNS
david@server:~$ sudo systemd-resolve --status
...
Link 2 (enp2s0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.0.1.1

[Master James (masterjames)]

Still a problem for Ubuntu 19.04 too
https://askubuntu.com/questions/1139536/ubuntu-19-04-bind-not-resolving-locally/1140151

[Tim Wiel (timwiel)]

A problem on 18.04 even with all updates installed

Comment #38 worked for me (well so far anyway)

[Clément Février (clement)]

I have a local DNS server resolved local IP address and it doesn't work since I migrate. My DHCP server is properly configured and my configuration worked out-of-the-box for years.

Jeff Carr in comment #32 said everything that needs to be said.

[Richard Pillay (richardpillay)]

Another vote to bump up the priority on this.

Just lost 2 days to this bug, mostly because the behavior is so stupidly borked that I never suspected that Ubuntu would be the cause. Can you imagine how stupid it is that the only machines on the network affected are the Ubuntu Desktops and Ubuntu Servers. Can you imagine how embarrassing it that even Windows machines follow the standards better than a Linux machine.

And the absolutely mind-boggling thing is that this is affecting every release from 16.04 to 19.04 and still it is not considered important enough to fix.

It is absolutely ridiculous that it would be considered more important to keep behavior that supports a few edge cases of software that wants to do their own DNS resolution in spite of the fact that it breaks the world-wide standards of DNS resolution and ignore DHCP-provided DNS addresses.

[George Davis C (georgejosephdavis)]

Is absolutely no one looking into this issue?

This issue has been with Ubuntu since 17.04 and we are in 19.04 at the moment. I have multiple Ubuntu server clusters and this issue is breaking every dns name resolution in all of them. I am unable to reliably collect logs and manage systems over the network.

Please bump the bug importance to Medium. DNS is an absolute necessity and nothing is more important than DNS if I have 100+ servers in a cluster but no networking.

I think it is obvious to anyone that this issue needs a resolution ASAP. Please address this issue with atmost priority.

[Vinicius (vinilanzoni)]

This affects me since version 18.04 LTS and the workaround by Vincent Fortier (th0ma7) on #8 solves it. But I'll be much thankful if this bug is fixed on the next LTS release (20.04). It impacts me on every production instance.
I also vote to change to "Insanely broken and wrong".

[ts (toto-23)]

Hi everybody,

upgrading to 19.10 the fix from #8 (thanks for helping with previous setups) does not seem to work anymore, as there's only a /run/resolvconf/resolv.conf (no stub-resolve anymore), which now keeps everything broken.

This leaves me with switching to another distro for the moment, as Ubuntu has stopped working in a sensible way at my office/professional setup.

Any hints would be highly appreciated on the short term, fixing systemd (or probably better: getting rid of it) on the medium term is definitely desirable (I do prefer debian/Ubuntu setups over Arch, etc.).

[FFO (fifo++)]

Problem exist with Ubuntu 19.10
The netplan configuration below just works
then after some time (I can't say when) it just stops resolving.
executing netplan apply make the DNS resolution work again.

----------------------------------------------
root@nas:/home/ff# nslookup www.cisco.com
Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find www.cisco.com: SERVFAIL

root@nas:/home/ff# netplan apply
root@nas:/home/ff# nslookup www.cisco.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
www.cisco.com canonical name = www.cisco.com.akadns.net.
www.cisco.com.akadns.net canonical name = wwwds.cisco.com.edgekey.net.
wwwds.cisco.com.edgekey.net canonical name = wwwds.cisco.com.edgekey.net.globalredir.akadns.net.
wwwds.cisco.com.edgekey.net.globalredir.akadns.net canonical name = e2867.dsca.akamaiedge.net.
Name: e2867.dsca.akamaiedge.net
Address: 104.69.84.242
Name: e2867.dsca.akamaiedge.net
Address: 2a02:26f0:e1:391::b33
Name: e2867.dsca.akamaiedge.net
Address: 2a02:26f0:e1:381::b33

----------------------------------------------

network:
    ethernets:
            eth0:
                    dhcp4: no
            eth1:
                    dhcp4: no
            eth2:
                    dhcp4: no
            eth3:
                    dhcp4: no
    bridges:
        br0:
            interfaces: [eth0, eth1, eth2, eth3]
            macaddress: 00:51:82:f1:f0:02
            dhcp4: no
            addresses: [192.168.0.32/24]
            gateway4: 192.168.0.2
            nameservers:
              addresses: [192.168.0.2]

    version: 2

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Francisco Sánchez-Aedo Gálvez (sanchezaedo)]

I just installed the 19.10 server iso and the machine doesn't use the DNS server provided through DHCP.

As a workaround I had to:
$ sudo rm /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

[Julian Petri (petroy)]

Thank you so much, this workaround has helped me fix the problem I described in another bug: https://bugs.launchpad.net/ubuntu/+source/network-manager-strongswan/+bug/1864256

[Steve Langasek (vorlon)]

This bug, as described, is about 127.0.0.53 being configured in /etc/resolv.conf /alongside/ other entries. In recent releases this is no longer the behavior; instead, /etc/resolv.conf is entirely managed by systemd and points exclusively to the 127.0.0.53 local resolver, which in turn will use the upstream resolvers that have been configured via dhcp or via static netplan or network-manager config.

Concerns about the behavior of the local resolver should be filed as separate issues.

[Jan Vlug (z-j)]

I have this issue as well. However, the instructions given in https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624320/comments/57 do not solve the issue for me.

When connecting the strongswan VPN, I see that the /run/systemd/resolve/resolv.conf file is touched (the timestamp of the file changes), but the content does not change.

Automatic nor manual DNS servers in the VPN settings do not work.

I tested manual modifying the /run/systemd/resolve/resolv.conf by changing the nameserver. This works once, but this change is not persisted, as the file is overwritten each time.

using fully up to date: Linux myhost 5.4.0-48-generic #52-Ubuntu SMP Thu Sep 10 10:58:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

[Jan Vlug (z-j)]

Note: The file /run/NetworkManager/no-stub-resolv.conf does contain the nameservers that are automatically provided by the VPN connection.

This solved the issue for me:
sudo rm /etc/resolv.d
sudo ln -s /run/NetworkManager/no-stub-resolv.conf /etc/resolv.conf

I am running Ubuntu Desktop, not Ubuntu Server (if that is relevant).

[Jan Vlug (z-j)]

Sorry, comment 60 and comment 61 were intended for bug 1864256.