systemd-resolved can't resolve at all, need to add nameservers to resolve.conf

[Jason Heeris]

I'm running Ubuntu 18.04 (upgraded from 17.10) on a machine with both ethernet and wifi interfaces. When I boot, my ethernet connection enp0s31f6 is brought up by Network Manager and given three nameserver addresses via DHCP, 10.1.13.10, 10.1.141.10, 10.1.13.36. Running nmcli shows the three nameservers under "DNS configuration". Running "systemd-resolve --status" shows them under a "Link 2 (enp0s31f6)" section. I can do a "ip route get to X" and ping each one successfully. No other connection is active.

testuser ☼ systemd-resolve --status
Global
          DNS Domain: orgsdomain.net
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp4s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (enp0s31f6)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 10.1.13.10
                      10.1.141.10
                      10.1.13.36
          DNS Domain: orgsdomain.net

However, when I actually try to resolve a name, even the name of one of the nameservers, dig claims that "connection timed out: no servers could be reached".

testuser ☼ dig dcpdc001.orgsdomain.net +nocookie

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> dcpdc001.orgsdomain.net +nocookie
;; global options: +cmd
;; connection timed out; no servers could be reached

Note that this name should resolve to 10.1.13.10, the first nameserver. The "+nocookie" option is there to work around an issue with Windows DNS servers. But other than that, the servers themselves work fine if I tell dig where to look:

testuser ☼ dig dcpdc001.orgsdomain.net +nocookie @10.1.13.10

; <<>> DiG 9.11.3-1ubuntu1.1-Ubuntu <<>> dcpdc001.orgsdomain.net +nocookie @10.1.13.10
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61294
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;dcpdc001.orgsdomain.net. IN A

;; ANSWER SECTION:
dcpdc001.orgsdomain.net. 3600 IN A 10.1.13.10

;; Query time: 2 msec
;; SERVER: 10.1.13.10#53(10.1.13.10)
;; WHEN: Fri Jul 20 10:56:27 AEST 2018
;; MSG SIZE rcvd: 65

I have configured resolvconf to use dynamic updates. /etc/resolv.conf points to /run/resolvconf/resolv.conf. This file contains only (non-comments):

nameserver 127.0.0.53
search orgsdomain

If I add "nameserver 10.1.13.10" to this file manually, suddenly dig can resolve again (without the @...), and anything else that needs to resolve names can do so. Removing the nameserver breaks that again.

I don't know much about the servers. They're part of a Windows-based network, but since I can use them if I edit resolv.conf or give dig the address, I don't think they're the issue (except in the sense that maybe they require a feature that systemd-resolved doesn't support?).

I increased the logging level of systemd-resolved to "debug" and "journalctl -f -u systemd-resolved" shows this during a failed query:

Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Got DNS stub UDP query packet for id 19836
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Looking up RR for dcpdc001.orgsdomain.net IN A.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Switching to DNS server 10.1.13.10 for interface enp0s31f6.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Cache miss for dcpdc001.orgsdomain.net IN A
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Transaction 12728 for <dcpdc001.orgsdomain.net IN A> scope dns on enp0s31f6/*.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Using feature level UDP+EDNS0+DO+LARGE for transaction 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Using DNS server 10.1.13.10 for transaction 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Sending query packet with id 12728.
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Processing query...
Jul 20 10:33:23 heerij-ubuntu systemd-resolved[2352]: Timeout reached on transaction 12728.

This repeats dozens of times, trying the other nameservers too. Note that there is substantially less than a second between the "Processing query..." message and the "Timeout reached..." message. (There are also problems with the other servers not having port 53 open, so I also get "Using degraded feature set..." messaged for them. But again, the first server seems to work fine with everything except systemd-resolved.)

Things I have tried:

  * Enabling DNSSEC. It's supported, but doesn't fix the issue.
  * Explicitly setting nameservers in Netplan's config. Is accepted, but doesn't change anything.

Sorry for the lack of the usual bug report attachments, but getting anything out of this machine and on to the internet is now a massive pain. Let me know what other debugging info might help and I'll try to add it.

[Jason Heeris (detly)]

Note also that upstream claim that "resolved is not supposed to be a DNS server"[1], so I'm a bit confused as to why Ubuntu is configured to pass every DNS query through it. Do I need to reconfigure something?

  [1] https://github.com/systemd/systemd/issues/4621#issuecomment-260050033

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michel-Ekimia (michel.ekimia)]

We seem to hit the same problem here.

Can you post you NetworkManager.conf file ? if it was not upgraded correctly that could be the issue

[Dmitry Lapshin (lapshin-dv)]

Any progress here? systemd-resolved has plus that it's integrated with DNS options from systemd-networkd, but for end user it has many issues, especially related to DNSSEC.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue