systemd-resolved appends 127.0.0.53 to resolv.conf alongside existing entries

Bug #1624320 reported by Anders Kaseorg on 2016-09-16
92
This bug affects 17 people
Affects Status Importance Assigned to Milestone
systemd (Ubuntu)
Low
Unassigned

Bug Description

systemd-resolved, or more precisely the hook script /lib/systemd/system/systemd-resolved.service.d/resolvconf.conf, causes resolvconf to add 127.0.0.53 to the set of nameservers in /etc/resolv.conf alongside the other nameservers. That makes no sense because systemd-resolved sets up 127.0.0.53 as a proxy for those other nameservers. The effect is similar to bug 1624071 but for applications doing their own DNS lookups. It breaks any DNSSEC validation that systemd-resolved tries to do; applications will failover to the other nameservers, bypassing validation failures. And it makes failing queries take twice as long.

/etc/resolv.conf should have only 127.0.0.53 when systemd-resolved is active.

Martin Pitt (pitti) wrote :

The primary purpose of adding 127.0.0.53 to resolv.conf is for client software that wants to do DNS resolution by itself instead of using NSS -- most notable example is Google Chrome, and third-party software which is statically linked (e. g. Go).

However, other software like NetworkManager or isc-dhcp also calls resolvconf and adds name servers picked up by them -- as they don't talk to resolved directly, resolved reads their DNS servers *from* resolv.conf.

But, software which does its own DNS lookups like the above have to do their own DNSSEC validation too -- you can't both chose to *not* use NSS *and* rely on NSS to do DNSSEC for you..

So, this is indeed a wart, but not easily fixed, and also not that important IMHO. Not using NSS is already broken to some degree, as you also ignore things like nss-{winbind,docker,ldap} etc.

Changed in systemd (Ubuntu):
status: New → Triaged
importance: Undecided → Low
Anders Kaseorg (andersk) wrote :

DNS resolution outside NSS shouldn’t be dismissed as an edge case for software that’s too conceited to use NSS. NSS only exposes A, AAAA, and PTR records. There’s plenty of software in the official archive that needs other records from DNS (off the top of my head: SRV, TXT, MX, SSHFP, AFSDB) and cannot get them from NSS.

> resolved reads their DNS servers *from* resolv.conf.

Right, so perhaps when resolvconf is in use, systemd-resolved should read those from /run/resolvconf/resolv.conf directly, and /etc/resolv.conf should be a symlink to /lib/systemd/resolv.conf rather than /run/resolvconf/resolv.conf?

> you can't both chose to *not* use NSS *and* rely on NSS to do DNSSEC for you.

Why not? It was working with NetworkManager managing a dnsmasq, since NetworkManager installed the local proxy as the only nameserver visible in resolv.conf, and it would work again if systemd-resolved did the same.

This will also be needed to fix problems like https://github.com/systemd/systemd/issues/3421 for programs that cannot use NSS.

Willem Toorop (willem-l) wrote :

Unfortunately the DNS interface of current systemd-resolved strips DNSSEC, so applications that do DANE validation still have to target the upstreams directly.
I have filed a bug about this: https://github.com/systemd/systemd/issues/4621

Martin Pitt (pitti) wrote :

As explained, 127.0.0.53 must be in /etc/resolv.conf in order to support Chromium and other software which does not NSS -- that is the whole raison d'être for providing a local stub server.

In Yakkety with NetworkManager only the 127.0.1.1 dnsmasq server is in /etc/resolv.conf, so that isn't affected. In Zesty NM now does not manager resolv.conf itself any more but just sends the acquired name servers to resolved.

Thus, is there anything left from the original bug report that still needs to be addressed?

Changed in systemd (Ubuntu):
status: Triaged → Incomplete
tags: added: resolved
Willem Toorop (willem-l) wrote :

Au contraire,

I argue to not fix this issue and keep the other upstream resolvers alongside 127.0.0.53 in /etc/resolv.conf, because 127.0.0.53 will break applications that need to do DNSSEC validation themselves (for example for DANE).

Once systemd-resolved has been fixed to provide the DNSSEC data alongside the answer, I agree with Anders that the other resolvers should go.

Anders Kaseorg (andersk) wrote :

Martin: I still wonder what the point of having resolvconf is, if it’s only ever supposed to be used to manage 127.0.0.53, and every other use of resolvconf will lead to this bug resurfacing. I still propose that systemd-resolved should read from /run/resolvconf/resolv.conf without adding 127.0.0.53 to it, and /etc/resolv.conf should be a symlink to /lib/systemd/resolv.conf rather than /run/resolvconf/resolv.conf.

Willem: If systemd-resolved isn’t suitable to be the only nameserver in resolv.conf, then it isn’t suitable to be in resolv.conf at all. I would rather either see these bugs worked out during the zesty cycle, or have systemd-resolved removed entirely, than leave the system in a half-broken state where the bugs in systemd-resolved get (probabilistically?) masked by the presence of other nameservers in resolv.conf.

In its current state, systemd-resolved is thoroughly broken on account of (at least) bug 1647031. I had to downgrade network-manager because switching to a resolver that doesn’t follow CNAME records breaks way too much of the internet. This might not have been noticed with other nameservers in resolv.conf. (Although now that it has, I’m getting increasingly concerned by the switch not having been reverted yet…)

Doug Goldstein (cardoe) wrote :

So this issue bit me in a weird way. I had my bridge called "xenbr0" and the result of these two interacting gives me absolutely no DNS resolution since the /etc/resolv.conf just pointed to 127.0.0.53 and didn't include the DNS server from my "xenbr0". Renaming the bridge to "renbr0" caused it to work. Why "renbr0"? Because in /run/resolvconf/interface/ there is a file called systemd-resolved that I was guessing was stopping combining the files once it hit the systemd-resolved.

Vincent Fortier (th0ma7) wrote :

From the man page systemd-resolve can run in 3 mode of operations.

1) Ubuntu 17.10 default - The default is to list the 127.0.0.53 DNS stub (see above) as only DNS server. This file may be symlinked from /etc/resolv.conf in order to connect all local clients that bypass local DNS APIs to systemd-resolved. This mode of operation is recommended.

2) WHAT YOU WANT: systemd-resolved maintains the /run/systemd/resolve/resolv.conf file for compatibility with traditional Linux programs. This file may be symlinked from /etc/resolv.conf and is always kept up-to-date, containing information about all known DNS servers....

3) PRE-17.04: Alternatively, /etc/resolv.conf may be managed by other packages, in which case systemd-resolved will read it for DNS configuration data. In this mode of operation systemd-resolved is consumer rather than provider of this configuration file.

The fix:
root@localhost:~# ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 29 mar 7 20:20 /etc/resolv.conf -> ../run/resolvconf/resolv.conf
root@localhost:~# rm -f /etc/resolv.conf
root@localhost:~# ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf
root@localhost:~# ls -la /etc/resolv.conf
lrwxrwxrwx 1 root root 32 mar 8 07:30 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

Finally firefox started working properly along with all my command line tools...

Leo Raczka (keseszel) wrote :

Hello everyone. Sorry for my english.
I have trouble with this f.. DNS, because 127.et around.
I change this for 8.8.8.8 - then FF working properly. I shut down my Ubuntu and DNS znowu 127.etc when I running Ubuntu. This sick.

Mike Loebl (mloebl) wrote :

I appreciate Vincent's post. I upgraded to 17.04 and DNS resolution has been absolutely terrible; slow, failed lookups, etc. Finally realized it was using 127.0.0.53 as the resolver. I did the method 2 above and DNS back to normal.

Tom Kidman (tom-kidman) wrote :

I had this problem too. I've upgraded 2 machines from kubuntu 16.10 to 17.04 and afterwards both were unable to resolve DNS. The only line in /etc/resolv.conf was 'nameserver 127.0.0.53'. As Mike wrote, following Vincent's fix above fixed the issue for me. Thanks Vincent :)

Fred Senior (freddyp) wrote :

Other *nix machines and Macs their resolve.conf on my LAN just append the dns server address that is configured om their network manager or network preferences. Why Ubuntu does not follow this I do not understand.

It stops my Ubuntu devices resolving FQDN's on the the LAN. I have to manually edit /etc/resolv.conf to fix this.

Same problem here. Editing resolv.conf or #8 post solves it.

I almost lost a RAID today because postfix was not able to send mdadm notifications -
delivery temporarily suspended: Host or domain name not found. Name service error for name=smtp.gmail.com type=MX: Host not found

I use /etc/network/interfaces, nothing in networkmanager.
This is serious. Ubuntu shouldn't just start changing things like this, specially if it breaks a server.

Using a clean 17.04 install here.

Alexander List (alexlist) wrote :

Using a clean 17.04 install, I could observe the same problem.

Against convention, our company is using .local as the internal DNS TLD.

As mentioned in #8 and #13 above, /etc/resolv.conf was symlinked to

root@dell-e5470:~# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 29 Apr 5 09:19 /etc/resolv.conf -> ../run/resolvconf/resolv.conf

which contains

---
# cat /run/resolvconf/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
---

After removing the symlink /etc/resolv.conf and linking like this:

# ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

...thinks started working again. Now the file contains the DNS received via DHCP:

-----
# cat /etc/resolv.conf
# This file is managed by systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known DNS servers.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 1.2.3.4
search local.example.com example.local
-----

Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
Dimitri John Ledkov (xnox) wrote :

@alexlist

At least for me, I do get

nameserver 127.0.0.53
search local.example.com example.local

In my resolv.conf by default. As in the exppected domains are listed in the search search stanza in the ../run/resolvconf/resolv.conf for my company network. As well as /run/systemd/resolv/resolv.conf.

Could you please check the following:
* move existing /etc/resolv.conf out of the way
* specify:
nameserver 127.0.0.53
search local.example.com example.local

And check that things work?

Also please do not test just the 17.04 clean install, but also upgrade systemd to the update shipped in zesty-updates first.

Dimitri John Ledkov (xnox) wrote :

If search stanzas are not generated for the 127.0.0.53 in resolv.conf, we need to open a new bug report I believe.

Otto (otto11235) wrote :

In my case after a 17.04 installation I needed a HOSTS file in order to resolve local hosts. After applying the fix in post #8 both local and remote hosts were resolved by the router DNS server (as was the case in the past).

Otto (otto11235) wrote :

Update to post #17...
The fix unfortunately did not fix all DNS problems. nslookup and dig can resolve local hosts, BUT ping and ssh both report 'Name or Service not known'. Looks like the HOSTS is still required.

Antoine Bertin (diaoulael) wrote :

I've been struggling with this issue... It seems I have to append the localdomain to my local names for the resolution to work :

nslookup somename : SERVFAIL, 127.0.0.53 didn't proxy to my router
nslookup somename.localdomain : OK, answer from my router

Why somename request isn't forwarded to my router?

Link 2 (eth0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.200
          DNS Domain: localdomain

I'm ok for some proxying to take place if that proves useful but so far it's only been a PITA for me so I disabled it following Vincent's answer.

Kevin Rudloff (krudloff) wrote :

Hello everyone! I had the same issue. Google Chrome and Firefox just loaded Google's websites.
I went to resolv.conf to see what was happening and I saw a new configuration. So I decided to add Google's DNS and it worked again but just till I restarted my laptop.
I don't know much of programming but the solution purposed by Vincent Fortier (th0ma7) worked for me. Thanks!

Bernhard (xro) wrote :

Hi,

I just upgraded from Ubuntu 16.10 to 17.04 and observed the following systemd-resolve behaviour that might be related to this bug:

--> Any domain listed after the "search" keyword in /etc/resolv.conf stops being resolved by systemd-resolve.

respectively

--> Any domain listed as "DNS Domain:" when querying "systemd-resolve --status" stops being resolved by systemd-resolve

I confirmed with tcpdump that systemd-resolve will not query my local or any dnsserver for any subdomains for any domain listed under the "search" keyword.

Changing the domains after the search keyword in /erc/resolv.conf immediately changes which domains systemd-resolve will ignore and not resolve.

The search domain is normally automatically set in response to receiving an "option domain-name" from the local dhcp server and normally used to translate requests for www to e.g. www.localdomain.at if localdomain.at was the domain-name sent by the DHCP server.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.