Unable to resolve local DNS names provided by router on bootup.

[b]

On boot, I can see systemd-resolve is configured properly; the router (192.168.1.254) is listed as a DNS server:

$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp0s20f3)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (eno1)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.254
          DNS Domain: telus

Local names are not resolved:

$ nslookup zf1

Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find zf1: SERVFAIL

$ dig zf1

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> zf1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 39908
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;zf1. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Feb 23 09:46:12 PST 2019
;; MSG SIZE rcvd: 32

Restarting netplan and network-manager services make no difference.

Restarting the systemd-resolved (or clearing systemd-resolved caches) solve the problem:

$ sudo systemd-resolve --flush-caches

OR

$ sudo service systemd-resolved restart

$ nslookup zf1
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: zf1.telus
Address: 192.168.1.71
** server can't find zf1.telus: NXDOMAIN

Dig still can't find server though:

$ dig zf1

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> zf1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 20375
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;zf1. IN A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Sat Feb 23 09:47:38 PST 2019
;; MSG SIZE rcvd: 32

Unless I specify the server manually:

$ dig @192.168.1.254 zf1
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.11.3-1ubuntu1.5-Ubuntu <<>> @192.168.1.254 zf1
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 50811
;; flags: qr; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 94b92f65104856d4 (echoed)
;; QUESTION SECTION:
;zf1. IN A

;; ADDITIONAL SECTION:
zf1. 10000 IN A 192.168.1.71

;; Query time: 6 msec
;; SERVER: 192.168.1.254#53(192.168.1.254)
;; WHEN: Sat Feb 23 09:59:05 PST 2019
;; MSG SIZE rcvd: 60

Once I get local names resolved, the systemd-resolve status remains the same:

$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (wlp0s20f3)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (eno1)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.254
          DNS Domain: telus

What is going wrong here?

This problem effects both my bonic machines, but does not happen on the machines still on xenial; I can thus assume the router is properly providing DNS and this indeed is a bionic bug.

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10.13
ProcVersionSignature: Ubuntu 4.15.0-45.48-generic 4.15.18
Uname: Linux 4.15.0-45-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.5
Architecture: amd64
CurrentDesktop: XFCE
Date: Sat Feb 23 09:49:29 2019
InstallationDate: Installed on 2019-02-09 (14 days ago)
InstallationMedia: Xubuntu 18.04.1 LTS "Bionic Beaver" - Release amd64 (20180725)
MachineType: Intel(R) Client Systems NUC8i3BEH
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-45-generic root=UUID=9c982b36-8142-4719-810a-e06f81cab223 ro quiet splash vt.handoff=1
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /lib/systemd/system/rc-local.service → /lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /lib/systemd/system/user@.service → /lib/systemd/system/user@.service.d/timeout.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 10/15/2018
dmi.bios.vendor: Intel Corp.
dmi.bios.version: BECFL357.86A.0051.2018.1015.1513
dmi.board.name: NUC8BEB
dmi.board.vendor: Intel Corporation
dmi.board.version: J72693-304
dmi.chassis.type: 3
dmi.chassis.vendor: Intel Corporation
dmi.chassis.version: 2.0
dmi.modalias: dmi:bvnIntelCorp.:bvrBECFL357.86A.0051.2018.1015.1513:bd10/15/2018:svnIntel(R)ClientSystems:pnNUC8i3BEH:pvrJ72753-303:rvnIntelCorporation:rnNUC8BEB:rvrJ72693-304:cvnIntelCorporation:ct3:cvr2.0:
dmi.product.family: Intel NUC
dmi.product.name: NUC8i3BEH
dmi.product.version: J72753-303
dmi.sys.vendor: Intel(R) Client Systems

[b (ben-ekran)]

Correction, service systemd-resolved restart does not solve the problem. Only systemd-resolve --flush-caches fixed the problem temporarily.

[b (ben-ekran)]

I noticed the following in my syslog recently:

systemd-resolved[656]: Server returned error NXDOMAIN, mitigating potential DNS violation DVE-2018-0001, retrying transaction with reduced feature level UDP.

This may be a hint.

Bug persists using systemd 237-3ubuntu10.24

[b (ben-ekran)]

Looking at this issue again I noticed some others having issues with /etc/resolv.conf so I check and indeed the symlink was incorrect:

/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

This was a fresh installation of bionic, so I'm not sure what package installed this incorrect symlink. The solution was to change the symlink:

/etc/resolv.conf -> ../run/systemd/resolve/resolv.conf

And now I don't need to flush caches for local names to be resolved.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue