libnss-mdns slow response

[Rumen]

mdns responses are very slow which degrades the preformance of some services in my lan due to this long delay. Please see the log below:

rumen@sirius:~/indigo.git$ time nslookup sirius.local
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: sirius.local
Address: 192.168.11.109
Name: sirius.local
Address: 172.17.0.1
Name: sirius.local
Address: fe80::7afb:484a:7307:cf6

real 0m5,224s
user 0m0,008s
sys 0m0,008s
rumen@sirius:~/indigo.git$ time nslookup dir.bg
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: dir.bg
Address: 194.145.63.12

real 0m0,035s
user 0m0,012s
sys 0m0,006s
rumen@sirius:~/indigo.git$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0
search www.tendawifi.com
rumen@sirius:~/indigo.git$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd
group: files systemd
shadow: files
gshadow: files

hosts: files mdns4_minimal [NOTFOUND=return] dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis
rumen@sirius:~/indigo.git$

[Trent Lloyd (lathiat)]

Rumen,

When you use 'nslookup' it should go directly to using the DNS server (127.0.0.53 [which is systemd-resolved]) which typically bypasses libnss-mdns but also typically doesn't have this 5 second delay (which avahi can have in some configurations). Seems most likely the 5 second delay is coming from inside systemd-resolved for some reason.

The best way to test with "NSS" is to use "getent hosts DOMAIN"

Could you please confirm the output of the following commands:

lsb_release -a

dpkg -l libnss-mdns

systemctl status avahi-daemon

time getent hosts sirius.local

time nslookup sirius.local # just to verify the problem still exists at the same time we do the above test

systemd-resolve --status --no-pager

- attach the file /etc/systemd/resolved.conf

[Rumen (rumengb)]

Hi, here is the information you need. By the way today the timeouts are much worse but still multiple of 5s...

rumen@sirius:~/indigo_imager.git$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04 LTS
Release: 20.04
Codename: focal
rumen@sirius:~/indigo_imager.git$ dpkg -l libnss-mdns
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=================-===============-============-============================================
ii libnss-mdns:amd64 0.14.1-1ubuntu1 amd64 NSS module for Multicast DNS name resolution
rumen@sirius:~/indigo_imager.git$ systemctl status avahi-daemon
● avahi-daemon.service - Avahi mDNS/DNS-SD Stack
     Loaded: loaded (/lib/systemd/system/avahi-daemon.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2020-07-20 09:44:01 EEST; 18min ago
TriggeredBy: ● avahi-daemon.socket
   Main PID: 859 (avahi-daemon)
     Status: "avahi-daemon 0.7 starting up."
      Tasks: 2 (limit: 9117)
     Memory: 1.9M
     CGroup: /system.slice/avahi-daemon.service
             ├─859 avahi-daemon: running [sirius.local]
             └─914 avahi-daemon: chroot helper

юли 20 09:47:45 sirius avahi-daemon[859]: Interface wlp3s0.IPv4 no longer relevant for mDNS.
юли 20 09:47:45 sirius avahi-daemon[859]: Leaving mDNS multicast group on interface wlp3s0.IPv4 with address 192.168.235.82.
юли 20 09:47:45 sirius avahi-daemon[859]: Withdrawing address record for fe80::d3e:1b47:1c0a:cc03 on wlp3s0.
юли 20 09:47:45 sirius avahi-daemon[859]: Withdrawing address record for 192.168.235.82 on wlp3s0.
юли 20 09:51:18 sirius avahi-daemon[859]: Joining mDNS multicast group on interface wlp3s0.IPv6 with address fe80::d3e:1b47:1c0a:cc03.
юли 20 09:51:18 sirius avahi-daemon[859]: New relevant interface wlp3s0.IPv6 for mDNS.
юли 20 09:51:18 sirius avahi-daemon[859]: Registering new address record for fe80::d3e:1b47:1c0a:cc03 on wlp3s0.*.
юли 20 09:51:18 sirius avahi-daemon[859]: Joining mDNS multicast group on interface wlp3s0.IPv4 with address 192.168.235.82.
юли 20 09:51:18 sirius avahi-daemon[859]: New relevant interface wlp3s0.IPv4 for mDNS.
юли 20 09:51:18 sirius avahi-daemon[859]: Registering new address record for 192.168.235.82 on wlp3s0.IPv4.
rumen@sirius:~/indigo_imager.git$ time getent hosts indigosky.local
192.168.235.1 indigosky.local

real 0m20,132s
user 0m0,000s
sys 0m0,005s
rumen@sirius:~/indigo_imager.git$ time nslookup indigosky.local
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: indigosky.local
Address: 127.0.1.1
;; connection timed out; no servers could be reached

real 0m15,015s
user 0m0,014s
sys 0m0,005s

rumen@sirius:~/indigo_imager.git$ time ping -c1 192.168.235.1
PING 192.168.235.1 (192.168.235.1) 56(84) bytes of data.
64 bytes from 192.168.235.1: icmp_seq=1 ttl=64 time=1.59 ms

--- 192.168.235.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.585/1.585/1.585/0.000 ms

real 0m0,00...

[Rumen (rumengb)]

Hi again,
This changes to the system config seems to partially fix the problem:

rumen@sirius:~/indigo_imager.git$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd
group: files systemd
shadow: files
gshadow: files

hosts: files mdns4 [NOTFOUND=return] dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis
rumen@sirius:~/indigo_imager.git$ cat /etc/mdns.allow
.local.
.local
rumen@sirius:~/indigo_imager.git$ time ping -c1 indigosky.local
PING indigosky.local (192.168.235.1) 56(84) bytes of data.
64 bytes from indigosky.local (192.168.235.1): icmp_seq=1 ttl=64 time=2.03 ms

--- indigosky.local ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 2.028/2.028/2.028/0.000 ms

real 0m0,008s
user 0m0,001s
sys 0m0,005s

All services work as expected, but still:

rumen@sirius:~/indigo_imager.git$ time getent hosts indigosky.local
192.168.235.1 indigosky.local

real 0m10,021s
user 0m0,000s
sys 0m0,005s

This may help debug the issue... I have one more Laptop with ubuntu 20.04 and the it has the same issue.

Thank you!

[Trent Lloyd (lathiat)]

Can you please confirm
(1) The timing of "getent hosts indigosky.local", "host indigosky.local", "nslookup indigosky.local" and "nslookup indigosky.local 192.168.235.1" all done at the same time (mainly adding the direct lookup through the server, wondering if nslookup is doing something weird in focal).
(2) The timings for the same if you switch mdns4 back to mdns4_minimal (but remove everything else)

[Rumen (rumengb)]

Here is the output, it does not work again even with the new config i am totally confused:

rumen@sirius:~$ time getent hosts indigosky.local
192.168.235.1 indigosky.local

real 0m10,121s
user 0m0,004s
sys 0m0,000s
rumen@sirius:~$ time host indigosky.local
indigosky.local has address 127.0.1.1
;; connection timed out; no servers could be reached

;; connection timed out; no servers could be reached

real 0m20,013s
user 0m0,014s
sys 0m0,005s
rumen@sirius:~$ time nslookup indigosky.local
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: indigosky.local
Address: 127.0.1.1
;; connection timed out; no servers could be reached

real 0m15,012s
user 0m0,011s
sys 0m0,007s
rumen@sirius:~$ time nslookup indigosky.local 192.168.235.1
Server: 192.168.235.1
Address: 192.168.235.1#53

Name: indigosky.local
Address: 127.0.1.1
** server can't find indigosky.local: REFUSED

real 0m0,019s
user 0m0,012s
sys 0m0,004s
rumen@sirius:~$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd
group: files systemd
shadow: files
gshadow: files

hosts: files mdns4 [NOTFOUND=return] dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis
rumen@sirius:~$ cat /etc/mdns.allow
.local.
.local

[Rumen (rumengb)]

And here is the output with the default config:

# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd
group: files systemd
shadow: files
gshadow: files

hosts: files mdns4_minimal [NOTFOUND=return] dns
networks: files

protocols: db files
services: db files
ethers: db files
rpc: db files

netgroup: nis
rumen@sirius:~$ time getent hosts indigosky.local
192.168.235.1 indigosky.local

real 0m20,151s
user 0m0,006s
sys 0m0,000s
rumen@sirius:~$ time host indigosky.local
indigosky.local has address 127.0.1.1
;; connection timed out; no servers could be reached

;; connection timed out; no servers could be reached

real 0m20,016s
user 0m0,012s
sys 0m0,008s
rumen@sirius:~$ time nslookup indigosky.local
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: indigosky.local
Address: 127.0.1.1
;; connection timed out; no servers could be reached

real 0m15,019s
user 0m0,013s
sys 0m0,008s
rumen@sirius:~$ time nslookup indigosky.local 192.168.235.1
Server: 192.168.235.1
Address: 192.168.235.1#53

Name: indigosky.local
Address: 127.0.1.1
** server can't find indigosky.local: REFUSED

real 0m0,018s
user 0m0,008s
sys 0m0,008s
rumen@sirius:~$

[Trent Lloyd (lathiat)]

This output is generally quite confusing.

Can you try remove the "search www.tendawifi.com" and see how it differs?

[Trent Lloyd (lathiat)]

ideally using mdns4_minimal specifically (or i guess, both, but generally not recommended to use mdns4 in most cases)

[Rumen (rumengb)]

It is auto generated.
On thisnetwork where I am now there is no such entry:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 127.0.0.53
options edns0

but the problem is the same.

On the bright side this changed the client and now it caches the resolved ip and connects with ip not the name... but this is an ugly patch and it fixes only one service. All other services are slow. That is interesting is that using mdns4 with mdns.allow fixes the issue sometimes not always. and sometimes it is even slower...

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Valentyna (valia0906)]

The problem still exists.