systemd-resolved leaks mDNS queries to DNS

[Sergio Callegari]

On a freshly installed ubuntu focal machine, access to local machines advertised via mDNS is now broken. This is a regression wrt eoan where it used to work.

Trying to ping something like <host>.local invariably results in pinging 40.68.249.35 because systemd-resolved passes the query to the DNS even if .local is reserved for multicast DNS and for some reasons <anything>.local seems to resolve to 40.68.249.35. This happens even if the avahi daemon is up and running.

Stopping systemd-resolved makes the mDNS resolution work as expected.

Not only this breaks standard workflows, but it also means that anyone pretending to be 40.68.249.35 on the network could probably impersonate any local host.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: systemd 245.4-4ubuntu3.1
ProcVersionSignature: Ubuntu 5.4.0-37.41-generic 5.4.41
Uname: Linux 5.4.0-37-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.2
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: KDE
Date: Tue Jun 16 23:43:22 2020
EcryptfsInUse: Yes
InstallationDate: Installed on 2020-02-16 (121 days ago)
InstallationMedia: Kubuntu 19.10 "Eoan Ermine" - Release amd64 (20191017)
MachineType: SCHENKER SCHENKER_SLIM14_SSL14L19
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.4.0-37-generic root=/dev/mapper/VG_NVMe-root ro quiet splash vt.handoff=7
SourcePackage: systemd
SystemdDelta:
 [EXTENDED] /usr/lib/systemd/system/rc-local.service → /usr/lib/systemd/system/rc-local.service.d/debian.conf
 [EXTENDED] /usr/lib/systemd/system/user@.service → /usr/lib/systemd/system/user@.service.d/timeout.conf

 2 overridden configuration files found.
UpgradeStatus: Upgraded to focal on 2020-05-23 (24 days ago)
dmi.bios.date: 10/02/2019
dmi.bios.vendor: INSYDE Corp.
dmi.bios.version: 1.07.04RTR1
dmi.board.asset.tag: Tag 12345
dmi.board.name: N141CU
dmi.board.vendor: SCHENKER
dmi.board.version: Not Applicable
dmi.chassis.asset.tag: No Asset Tag
dmi.chassis.type: 10
dmi.chassis.vendor: Notebook
dmi.chassis.version: N/A
dmi.modalias: dmi:bvnINSYDECorp.:bvr1.07.04RTR1:bd10/02/2019:svnSCHENKER:pnSCHENKER_SLIM14_SSL14L19:pvrNotApplicable:rvnSCHENKER:rnN141CU:rvrNotApplicable:cvnNotebook:ct10:cvrN/A:
dmi.product.family: Not Applicable
dmi.product.name: SCHENKER_SLIM14_SSL14L19
dmi.product.sku: Not Applicable
dmi.product.version: Not Applicable
dmi.sys.vendor: SCHENKER

[Sergio Callegari (callegar)]

Did some more checks about this. If I understand correctly:

- Passing ".local" queries to the DNS is something that now happens because systemd-resolved is doing all of the name resolution (DNS and mDNS) while in the past it was the avahi daemon to be in charge of the mDNS resolution.

- Specifically, the reason is that systemd-resolved adds its own switches to configure mDNS and by default it is configured to pass ".local" queries to the DNS.

- You can have systemd-resolved use mDNS only for .local names by either
a) activating DNSSEC in its global configuration; or
b) activating multicast DNS in its global configuration and then tweak the Network Manager configuration files for the individual connections by adding mdns=2 in the [connection] section.

I have some comments/inquires on this:

- the Network Manager configuration with respect to mdns is not something that can be done from the GUI (at least with the KDE applet), which makes recovering the "normal" mDNS behavior inconvenient;

- the implications of activating DNSSEC in systemd-resolved are unclear to me. Is everything ready to do this without side effects?

- It is my understanding that the reason why mDNS is now passed by default to regular DNS is that in some enterprise environments the .local domain is used as a local domain controlled by an internal DNS. However, again to my understanding, this should not be the standard according to RFC 6762. Even less so now that the operation of many consumer network-attached peripherals is based on zeroconf. Because of this, I think that the default should be the opposite and that if .local needs to be resolved by unicast DNS, then in that case a configuration action should be taken.

- Even in environments where hosts on the .local domain require unicast DNS, it seems unreasonable to attempt that if the configured DNS is not local (e.g., the ISP provided one, google, etc.). In fact it is unclear to me why on public DNS *.local points to 40.68.249.35. Probably my ignorance, found no reference to it googling.

- It is unclear to me why systemd-resolved places additional configuration switches in front of mDNS when this should be already configured in nsswitch.conf. Particularly the default nsswitch.conf says to lookup first using mDNS, so when this is not done the result is confusing.

- Ubuntu seems to ship by default with both systemd-resolved and the avahi-daemon. Is this redundant? Is there a way to tell systemd-resolved to refrain from mDNS lookup and just hand it over to the avahi daemon? Shouldn't systemd-resolved just do that when mDNS is disabled for it but globally enabled via nsswitch? Is the avahi daemon needed at all if systemd-resolved can take care of everything? Can there be issues in having both up? For instance the arch wiki explicitly says "If Avahi has been installed, consider disabling avahi-daemon.service and avahi-daemon.socket to prevent conflicts with systemd-resolved".

[Sergio Callegari (callegar)]

Again, more analysis...

looks like the 40.68.249.35 is an answer delivered by my ISP that is hijacking DNS requests on port 53 (redirecting the request to a server of them even if I am specifying another server by IP address). For instance

host -t A pippo.pertica.palla 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:

pippo.pertica.palla has address 40.68.249.35

This is easily verified because if I reconfigure systemd-resolved to access the cloudflare dns 1.1.1.1 via TLS the DNS highjacking cannot happen anymore.

Hence apparently nothing may have changed in ubuntu and systemd-resolved and the issue presented here may be no regression.

Still, IMHO there is an issue with systemd-resolved. For .local hosts it seems to first query unicast DNS and only over a resolution failure to let the avahi-daemon do the mDNS resolution. However this is just the opposite of what my nsswitch.conf specifies (first query mDNS, then dns).

[Marc Deslauriers (mdeslaur)]

Hi,

Can I make this bug public?

[Sergio Callegari (callegar)]

On my side, certainly so.

[Marc Deslauriers (mdeslaur)]

Out of curiosity, what does the hosts line in your /etc/nsswitch.conf file look like?

[Sergio Callegari (callegar)]

Host entry in nsswitch.conf is:

files mdns4_minimal [NOTFOUND=return] dns

Note that I do not see the issue anymore, because the ISP has (for the time being) stopped messing so aggressively with dns.

In any case, I've moved to using dns over tls which systemd.resolved supports perfectly with 1.1.1.1 or 9.9.9.9, which should prevent the isp from troubling me even if they restart doing weird things (wonder if it was some sort of experiment on their side).

Nonetheless, I believe that the system should avoid breaking .local resolution and obey the priorities in the nsswitch file.

[Sergio Callegari (callegar)]

Finished my investigation. Guess it is not a bug after all, rather — I would say — a matter of extremely dispersed documentation plus very unfriendly behavior from the ISP.

The situation is that even if you put mdns_minimal in front of other entries in hosts in nsswitch, mdns /is not queried/ if the DNS declares SOA for the "local" domain. Mine does. This is the reason why I was getting the impression of systemd-resolved passing queries to DNS even when it should not have. It was actually nss_mdns to let it do so. The fact that for some time my isp has directed all dns queries to its own nameserver even when another one was selected did not help in verifying the behavior with a dns not declaring SOA for local.

Yet, this aspect of the linux mdns implementation should be documented with much better emphasis, or even better, there should be a way to make mdns_minimal report/log what it is doing. Currently, you find mention of the mdns "handover" to dns for local only looking at the very end of /usr/share/doc/libnss-mdns/README.md.gz where the heuristics is explained.

Because heuristics /can go wrong/, the item should IMHO be given much better emphasis. For instance, I suggest that ubuntu ships with a man page for nss_mdns and nss_mdns_minimal.