systemd-resolved.service still breaks (now 20.04, since as least 16)

[Gerben]

There are still occurrences of this service breaking down.

At some moment the service just stops working correctly and is not able to recover.

I can reset the service by restarting it:
   systemctl restart systemd-resolved.service

I've also tried to set the caching off, but it still fails on me occasionally (on 18.04 as well).

There is a local DNS in the network, I would like to directly use that in stead of:
   nameserver 127.0.0.53

Example:
geus@desk-02:~$ ssh smb-01
ssh: Could not resolve hostname smb-01: Temporary failure in name resolution
...
root@desk-02:/home/geus# systemctl restart systemd-resolved.service
root@desk-02:/home/geus# nslookup smb-01
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: smb-01.lan
Address: 192.168.178.39
...
geus@desk-02:~$ ssh smb-01
geus@smb-01's password:

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd 245.4-4ubuntu3.2
ProcVersionSignature: Ubuntu 5.4.0-42.46-generic 5.4.44
Uname: Linux 5.4.0-42-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.6
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: GNOME
Date: Sat Aug 8 17:03:50 2020
InstallationDate: Installed on 2020-05-31 (69 days ago)
InstallationMedia: Ubuntu 20.04 LTS "Focal Fossa" - Release amd64 (20200423)
Lsusb:
 Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd QEMU USB Tablet
 Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
 Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
 Bus 002 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Lsusb-t:
 /: Bus 04.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 03.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=uhci_hcd/2p, 12M
 /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=ehci-pci/6p, 480M
     |__ Port 1: Dev 2, If 0, Class=Human Interface Device, Driver=usbhid, 480M
MachineType: QEMU Standard PC (i440FX + PIIX, 1996)
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-42-generic root=/dev/mapper/vgubuntu-root ro quiet splash vt.handoff=7
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /usr/lib/systemd/system/rc-local.service → /usr/lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /usr/lib/systemd/system/user@.service → /usr/lib/systemd/system/user@.service.d/timeout.conf

 2 overridden configuration files found.
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 04/01/2014
dmi.bios.vendor: SeaBIOS
dmi.bios.version: 1.10.2-1ubuntu1
dmi.chassis.type: 1
dmi.chassis.vendor: QEMU
dmi.chassis.version: pc-i440fx-bionic
dmi.modalias: dmi:bvnSeaBIOS:bvr1.10.2-1ubuntu1:bd04/01/2014:svnQEMU:pnStandardPC(i440FX+PIIX,1996):pvrpc-i440fx-bionic:cvnQEMU:ct1:cvrpc-i440fx-bionic:
dmi.product.name: Standard PC (i440FX + PIIX, 1996)
dmi.product.version: pc-i440fx-bionic
dmi.sys.vendor: QEMU

[Dan Streetman (ddstreet)]

To debug the systemd-resolved service, you can run

$ sudo systemctl edit systemd-resolved

and in the editor that is opened, add and save this content:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

then restart systemd-resolved, or reboot. Then once the problem happens again, check:

$ journalctl -b -u systemd-resolved

to see if there are any problems logged. You can check the current log also, as it may have logged some problems even without debugging enabled, although to check the logs for previous boots you should leave the -b parameter off.

If any problem was logged you can paste that part of the log here, or just attach the full log.

[Gerben (gerbgeus)]

I've now got one instance with this condition again.

From what I can see from the logs is that systemd-resolved started using some other DNS server:
First one in my network where I do not have control over (ipv4: 192.168.178.1 , somehow it turned up with this fairly correct address : my router)
Then one which I do not know at all (ipv6: fd00::464e:6dff:fefb:1b50 , don't know where that is)

What I would like is to have it stick to 192.168.178.2 , which is used to supply addresses to virtual machines.

I'll leave systemd-resolved running as is for now for inspection.

Aug 18 12:09:38 kvm1804 systemd-resolved[2418]: Switching to DNS server 192.168.178.1 for interface br0.
...
Aug 18 12:56:15 kvm1804 systemd-resolved[2418]: Switching to DNS server fd00::464e:6dff:fefb:1b50 for interface br0.

[Gerben (gerbgeus)]

192.168.178.1 was defined as secondary nameserver on kvm1804 within 01-netcfg.yaml (ubuntu 18.04).

The primary nameserver is just operating properly. I do not see any reason why systemd-resolved would not target this primary nameserver, let alone target something external to me at all.

In case of emergency it might use a subsequent nameserver, but it should switch back when possible.

To my knowledge the primary nameserver did not have any problems during these switches.

This machine has a static ip address, it is a physical machine running virtuals.

I've seen "Switching to DNS server" logged on desk-02 (ubuntu 20.04.1) for which I filed this bug.
This directly switched to the external ipv6 fd00::464e:6dff:fefb:1b50.

[Gerben (gerbgeus)]

For completeness, 01-netcfg.yaml for machine kvm1804

[Gerben (gerbgeus)]

I think I'll steer away from systemd-resolved

1) it does not favor my primary nameserver (after a switch it'll not switch back, but continue any failover server)
2) in the end it can even use a nameserver which I have not provided (which is set by the packagebuilder)

Reading into some threads on systemd-resolved it is clear all nameservers are expected equal. This is clearly not the case with small business (or any business, if you ask me).

The uncontrolled usage of the external nameserver set by the packagebuilder is also a weird choice.

These choices might be OK in certain situations (e.g. a laptop to access the web for the wife), but clearly fails my needs.

To change to something else, seel (e.g.):
      https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

(probably this is the cause of some nameserver issue I reported on docker, where docker clients ended up using 8.8.8.8)

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]