systemd-resolved’s 127.0.0.53 server does not follow CNAME records

[Anders Kaseorg]

[SRU Justification]
Ubuntu 16.10 server uses systemd-resolved by default, configured both as a DNS stub resolver on 127.0.0.53 and as an NSS module via libnss-resolved talking to the dbus service. The DNS stub resolver has a bug that causes it to fail to resolve CNAME records. This went unnoticed before release because by default the NSS module is used. But a chroot or container on the system that does not include libnss-resolved and is configured to use the stub resolver will experience DNS failures.

[Test case]
1. On a yakkety server system, create a xenial chroot with mk-sbuild (or equivalent).
2. Make sure that the host system has /etc/resolv.conf pointed at 127.0.0.53.
2. Enter the chroot with 'sudo schroot -c xenial-amd64' or such.
3. Install the iputils-ping package.
4. ping www.freedesktop.org
5. Confirm that the hostname does not resolve.
6. Install the systemd package from yakkety-proposed onto the host system.
7. ping www.freedesktop.org
8. Confirm that the hostname does now resolve.

[Regression potential]
With a 247-line patch to a key service, there is some risk of regression. Regression risk is mitigated because this patch is already present in zesty and upstream, where no regressions have been reported, and because it only touches the DNS stub resolver which is not the code path used by default on host systems.

$ systemd-resolve www.freedesktop.org
www.freedesktop.org: 131.252.210.176
                     2610:10:20:722:a800:ff:feda:470f
                     (annarchy.freedesktop.org)

-- Information acquired via protocol DNS in 673.6ms.
-- Data is authenticated: no
$ ping www.freedesktop.org
ping: www.freedesktop.org: Name or service not known
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
# 127.0.0.53 is the systemd-resolved stub resolver.
# run "systemd-resolve --status" to see details about the actual nameservers.

nameserver 127.0.0.53
$ dig +no{cmd,comments,stats} www.freedesktop.org @127.0.0.53
;www.freedesktop.org. IN A
www.freedesktop.org. 7146 IN CNAME annarchy.freedesktop.org.
$ dig +no{cmd,comments,stats} www.freedesktop.org @8.8.8.8
;www.freedesktop.org. IN A
www.freedesktop.org. 14399 IN CNAME annarchy.freedesktop.org.
annarchy.freedesktop.org. 14399 IN A 131.252.210.176

I trust it needn’t be explained why this makes the internet almost completely useless in zesty.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Martin Pitt (pitti)]

I confirm the fact that "dig @127.0.0.53 wiki.freedesktop.org" only gives the CNAME response, not the resolution of "annarchy.freedesktop.org." as well, which is sufficient to confirm the fix.

But nevertheless, firefox, wget, ping etc. on wiki.freedesktop.org all work fine, but these use NSS. Google Chrome also works fine (which uses resolv.conf), so this appears to be more like an implementation detail than a major user-visible bug to me? Can you confirm this, or am I missing something?

[Anders Kaseorg (andersk)]

Firefox, wget, ping, and Chrome do not work unless libnss-resolve is installed. (Perhaps Chrome recently started using NSS in some cases?)

Maybe this isn’t quite as critical as I thought given that ubuntu-standard Recommends libnss-resolve. Still, it’s not just an “implementation detail”. DNS lookups that cannot go through NSS (e.g. SRV/TXT/MX/SSHFP/AFSDB records or any program that needs an asynchronous API) are broken with or without libnss-resolve installed. And nothing pulls in libnss-resolve:i386 on amd64.

[Louis Bouchard (louis)]

I have libnss-resolve installed and still cannot connect to gmail.google.com with 127.0.0.53.

Happy to help diagnose the issue though.

[Martin Pitt (pitti)]

@Anders: ah, so you removed libnss-resolve, but manually enabled systemd-resolved.service?

[Anders Kaseorg (andersk)]

Whether or not it’s enabled, it is started automatically via D-BUS activation (/lib/systemd/system/dbus-org.freedesktop.resolve1.service -> systemd-resolved.service).

[Anders Kaseorg (andersk)]

Another case that now necessarily relies on 127.0.0.53 is anything inside a chroot (e.g. sbuild). libnss-resolve can’t help there: you can’t talk to the host D-Bus from inside, and also you might want to chroot a distro that’s too old for libnss-resolve anyway.

[Anders Kaseorg (andersk)]

Or virtual machines: a VM running in virt-manager on a zesty host can’t resolve CNAMEs (not even a zesty VM with libnss-resolve inside!).

[Steve Langasek (vorlon)]

Hi Anders,

This is reported as an issue in zesty, but I'm seeing the same behavior with resolved in yakkety with that same configuration. Did you have reason to believe this was a new problem only in zesty?

[Chris Halse Rogers (raof)]

This also breaks network connectivity in lxd containers; amusingly, resolving parts.snapcraft.io requires following the CNAME record, so “snapcraft cleanbuild” is broken on zenial unless NetworkManager is downgraded to the version before the systemd-resolved switch.

[Anders Kaseorg (andersk)]

Steve: I didn’t test yakkety since I thought it had gone back to the configuration with dnsmasq on 127.0.0.1 rather than systemd-resolved on 127.0.0.53. If that’s not the case, then yeah, it would be affected too.

[Barry Warsaw (barry)]

I just hit this same problem, as described here: https://lists.ubuntu.com/archives/ubuntu-devel/2017-January/039623.html

[Barry Warsaw (barry)]

I will test the revert of this change until upstream fixes their bug.

[Barry Warsaw (barry)]

As per pitti's email in #12, added network-manager bugtask and subscribed pitti.

[Barry Warsaw (barry)]

I've uploaded a version with the reverts to my PPA:

https://launchpad.net/~barry/+archive/ubuntu/experimental

A local build looks promising, but I want to test it more before I upload it for real. Please do test the PPA version once it builds.

[Barry Warsaw (barry)]

I've tested the PPA package and am satisfied that it restores CNAME resolution in schroots without any observed regression otherwise. I will upload it momentarily.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager - 1.4.2-3ubuntu2

---------------
network-manager (1.4.2-3ubuntu2) zesty; urgency=medium

  * Revert 98974a88 and 47c16e59 (network-manager switching from dnsmasq
    to resolved) because of bugs in resolved's resolution of CNAME
    records. (LP: #1647031)

 -- Barry Warsaw <email address hidden> Thu, 05 Jan 2017 15:26:55 -0500

[Martin Pitt (pitti)]

Fixed upstream: https://github.com/systemd/systemd/commit/e8d23f92b50a97bb3

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 232-17ubuntu1

---------------
systemd (232-17ubuntu1) zesty; urgency=medium

  * debian/patches/0001-resolved-follow-CNAMES-for-DNS-stub-
    replies.patch: cherry-pick upstream fix for following CNAMEs in DNS
    stub replies. Closes LP: #1647031.

 -- Steve Langasek <email address hidden> Sun, 12 Feb 2017 22:54:55 -0800

[Barry Warsaw (barry)]

I'll cherry pick back 98974a88 and 47c16359 now.

[André (afsverissimo)]

version 1.4.4-1ubuntu2 (from pre-release and from Barry's ppa) is not working correctly for me

Can only ping any host by ip and never by hostname (google.com / sapo.pt)

[André (afsverissimo)]

output of

$ systemd-resolve --status

[Fred (eldmannen+launchpad)]

Barry, I think something went wrong. My DNS worked previously, but after the update it resolves google.com but not many other domains such as youtube.com. :(

[Barry Warsaw (barry)]

Interesting. It worked and works for me. I tested the version in my PPA, both outside and inside a chroot (it was the inside-chroot that was previously broken). I just dist-upgraded my other Zesty machine and rebooted, and I am having no troubles resolving both IP and domain names both on my LAN and in the wider internet (www.ubuntu.com, www.python.org, youtube.com). The output of `systemd-resolve --status` LGTM, with my bridge0 link getting the correct DNS server (local to my LAN) and DNS domain.

chroots also look good (previously they could not get to my apt-cache-ng proxy on my LAN).

Virtual machines on the same host are also working fine afaict.

Since two of you have reported problems so closely together, there must be Something Going On, but I cannot reproduce it.

Can you please review your Network Connections from the network manager settings panel? Is there anything weird, misconfigured, or unexpected there?

[Steve Langasek (vorlon)]

André, Fred, are you both definitely running the newest version of systemd-resolved? You must be using the version from systemd 232-17ubuntu1, which just reached zesty yesterday. The resolved service *should* be automatically restarted when you upgrade the package.

[Fred (eldmannen+launchpad)]

Was chatting then I noticed my DNS had suddenly started working normally and YouTube resolved again.

I ran systemd-resolve --status
Under "Link 2 (eth0)" it said "DNSSEC supported: no".

I then restarted the computer, and now the DNS is broken again. It now says "DNSSEC supported: yes".

Minutes later (perhaps coincidentally after I started Chromium for the second time?) my DNS magically works again, and systemd-resolve reports "DNSSEC supported: no".

It seems DNS does not work when systemd-resolve reports "DNSSEC supported: yes".
And it seems that DNSSEC changes to no and starts working for some reason, I am not sure if this is triggered by time or some user action such as starting a browser or something.

[Fred (eldmannen+launchpad)]

Steve, I am using 232-18ubuntu1.

$ dpkg -l | grep systemd
ii dbus-user-session 1.10.10-1ubuntu2 all simple interprocess messaging system (systemd --user integration)
ii libnss-resolve:amd64 232-18ubuntu1 amd64 nss module to resolve names via systemd-resolved
ii libpam-systemd:amd64 232-18ubuntu1 amd64 system and service manager - PAM module
ii libpam-systemd:i386 232-18ubuntu1 i386 system and service manager - PAM module
ii libsystemd0:amd64 232-18ubuntu1 amd64 systemd utility library
ii libsystemd0:i386 232-18ubuntu1 i386 systemd utility library
ii python3-systemd 233-1 amd64 Python 3 bindings for systemd
ii systemd 232-18ubuntu1 amd64 system and service manager
ii systemd-sysv 232-18ubuntu1 amd64 system and service manager - SysV links

[Fred (eldmannen+launchpad)]

So when I start my computer, then I cannot ping youtube.com, I start Chromium, I still cannot ping youtube.com, then I open Firefox, and boom magically everything worked! :)

[Barry Warsaw (barry)]

I will say that I've noticed a delay in having the network come up too, but that predates the new network-manage, and the latest systemd. It's been going on for a while now but I haven't managed to report a bug on it yet.

I have several startup apps configured in my Gnome environment, including Chromium, Emacs, Claws, and Firefox (yes, two browsers :). Immediately after a reboot and login, once say Chromium comes up, many of the pages are reporting errors that they can't get to the domain. Just visiting the tab is enough to trigger Chromium to do an auto-reload of the page, and that always works. Or if I have to manually reload, it works. There's just some small bit of time after a reboot+login where the network is inaccessible.

But I don't think this is at all related to this bug. I suppose I should try to figure out which package is at fault and report a bug on it.

[Martin Pitt (pitti)]

Note: We keep DNSSEC=allow-downgrade during development to collect feedback, but switch it off for stable releases (we did so in yakkety and should do so again in zesty). So if you have some trouble which is DNSSEC related, it would be good to get a debug output of resolved while it's failing to report/investigate it. But that should be unrelated to the CNAME bug here.

[Khurshid Alam (khurshid-alam)]

Same problem as Fred here. It worked before the cherry-picking back 98974a88 and 47c16e59. But now it's broken. And yes everything works with "DNSSEC supported: no".

@Martin

How do I disable DNSSEC? Through /etc/systemd/resolv.conf? And how to switch it off "DNSSEC=allow-downgrade" ?

[Simplehuman (simplehuman)]

#31, same problem also. As workaround for me - I just downgraded network-manager package to 1.4.4-1ubuntu1 . Downloaded it from here - https://launchpad.net/ubuntu/+source/network-manager/1.4.4-1ubuntu1/ and then sudo dpkg -i network-manager_1.4.4-1ubuntu1_amd64.deb . And blocked this version in Synaptic. Probably not the best solution, but the best that I could find for now and it works.
Bug is really critical for dayly use.

[Martin Pitt (pitti)]

Yes, there, see "man resolved.conf". But I'd recommend a separate file to avoid changing the package-provided conffile:

sudo mkdir -p /etc/systemd/resolved.conf.d
printf "[Resolve]\nDNSSEC=no\n" | sudo tee /etc/systemd/resolved.conf.d/no-dnssec.conf

[Steve Langasek (vorlon)]

As I recall resolved is also enabled by default in Ubuntu Server 16.10 (though not on Desktop), so this is a critical issue there as well. Dimitri, could you please have a look at this backport?

Could someone who's seeing the DNSSEC problem please also file a separate bug report, so we can track that problem separately?

[Steve Langasek (vorlon)]

I was reminded that libnss-resolve is in 'standard' now, which means it's also installed by default on server and any bugs that affect only the stub DNS resolver without affecting the dbus service - such as this one - do not impact DNS resolution by default on server in 16.10.

(I was wondering why no one had reported it before now.)

So, I'm downgrading this from critical to high since it only hits users of 16.10 server in a few corner cases (i.e. containers, chroots without libnss-resolve installed). It should still be SRUed.

[Blaisorblade (p-giarrusso)]

Replying to #35:
> So, I'm downgrading this from critical to high since it only hits users of 16.10 server in a few corner cases (i.e. containers, chroots without libnss-resolve installed). It should still be SRUed.

Another corner case seems to be binaries linked against musl libc, since they do not use NSS.

We're getting many reports related the problem on the Haskell stack tool (https://github.com/commercialhaskell/stack/issues/2536#issuecomment-285327722), and we indeed link against musl libc (https://github.com/commercialhaskell/stack/issues/3060).

To be sure, is the plan to make the local DNS proxy at least resolve CNAME correctly on Yakkety and future releases, either by fixing systemd or switching to dnsmasq? Only providing `libnss-resolve` is not enough. I'm not aware of us needing fancier DNS features, but correct CNAME support would be great.

I understand you don't include musl libc, but except for this bug it's an attractive option for shipping one universal Linux binary, which I suggest Ubuntu should keep supporting. Since the bug affects other scenarios anyway, I think it's reasonable to hope for a fix. I appreciate your effort.

[Martin Pitt (pitti)]

Blaisorblade [2017-03-15 15:03 -0000]:
> Another corner case seems to be binaries linked against musl libc, since
> they do not use NSS.

Note that this is generally broken and cannot be supported, regardless of the
DNS resolver. These binaries could also not resolve winbind host names, YP,
LDAP, Active Directory, or sssd users/groups, and anything else which is
handled through nsswitch.

[Khurshid Alam (khurshid-alam)]

I don't know if this is related but now it simply could not resolve https://www.aviationweather.gov and www.noaa.gov. The result is the same through ping and wget.

However when I try ip address of www.aviationweather.gov in browser it works.

-------------------------------------------------------------------------------
$ systemd-resolve 140.90.101.207
140.90.101.207: resolve call failed: Address '140.90.101.207' does not have any RR of requested type

$ wget "https://140.90.101.207/adds/dataserver_current/httpparam"

Connecting to 140.90.101.207:443... connected.
    ERROR: certificate common name ‘ncep.noaa.gov’ doesn't match requested host name ‘140.90.101.207’
-------------------------------------------------------------------------------

I have libnssresolve installed. systemd-232-19, Ubuntu 17.04

[Khurshid Alam (khurshid-alam)]

Ah. Sorry. libnss-resolve was indeed the culprit. Removing it fixed the issue.

[Khurshid Alam (khurshid-alam)]

And the problem is back in next reboot. Not sure what is happening here.

[Łukasz Zemczak (sil2100)]

Hello Anders, or anyone else affected,

Accepted systemd into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/231-9ubuntu4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Sudeep Duggal (duggalsudeep-deactivatedaccount)]

Hello,

I did a clean install of the Ubuntu 17.04 release today. All websites were opening correctly in Chrome/Firefox using on a LAN connection. However, on WiFi, gmail/youtube are not opening in Chrome/Firefox. I am able to successfully ping both gmail and youtube. Other websites successfully open.

I ran "systemd-resolve --status" with "DNSSEC supported: yes". After multiple reboots and tries, youtube.com opened in the browser. However, I ran the above command and got "DNSSEC supported: no".

I am now getting "DNSSEC supported: yes" and gmail / youtube are not opening in the browser.

Is there any update on this bug ?

Regards

[Anders Kaseorg (andersk)]

Martin, was the DNSSEC=allow-downgrade default not switched off for the 17.04 stable release like you said it would?

[Jeremy Bícha (jbicha)]

Anders, this commit was pushed to stretch but not to zesty:

https://anonscm.debian.org/git/pkg-systemd/systemd.git/commit/?h=stretch&id=3361c8f55c

[Eldar Khayrullin (eldar)]

The same issue as Sudeep Duggal say (Ubuntu 17.04 bug, Ubuntu 16.10 works fine).

[Steve Langasek (vorlon)]

This bug report is about resolved not resolving CNAME records. If you are seeing issues with resolved and DNSSEC, please follow LP: #1682499 instead.

[Steve Langasek (vorlon)]

Verified by way of a slightly modified test case:
 - boot a yakkety cloud image
 - verify 127.0.0.53 in /etc/resolv.conf
 - sudo sed -i -e's/resolve \[[^]]*\] //' /etc/nsswitch.conf
 - ping www.freedesktop.org -> FAIL
 - install systemd from -proposed
 - ping www.freedesktop.org -> SUCCESS

[Steve Langasek (vorlon)]

Ubuntu Core 16 also uses resolved, so this bug also needs to be fixed in xenial.

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 231-9ubuntu4

---------------
systemd (231-9ubuntu4) yakkety; urgency=medium

  * debian/extra/units/systemd-resolved.service.d/resolvconf.conf: if
    resolved is going to be started, make sure this blocks
    network-online.target. LP: #1673860.
  * debian/patches/resolved-follow-CNAMES-for-DNS-stub-replies.patch:
    Cherry-pick upstream fix for resolved failing to follow CNAMES for DNS
    stub replies. LP: #1647031.
  * debian/patches/logind-update-empty-and-infinity-handling-for-User-T.patch:
    Cherry-pick upstream fix to handle empty and "infinity" values for
    [User]TasksMax. Closes LP: #1651518.

 -- Steve Langasek <email address hidden> Mon, 20 Mar 2017 22:14:14 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dimitri John Ledkov (xnox)]

127.0.0.53 stub resolver was only introduced in v231, thus there is nothing to fix in xenial.