systemd resolves own hostname to link local ipv6 address

[Andreas L]

I've got an ethernet-device that only has a configured ipv4 address, and some auto-generated link-local (aka "scope link") ipv6 address.

Any tool doing a DNS query (and /lib/systemd/systemd-resolved is the DNS-server listening on 127.0.0.53) for this host's hostname gets back two addresses: the correct ipv4 address, and a broken ipv6 address.

Unlike on ipv4, it is possible for the same ipv6-address to be assigned to multiple devices, and therefore the address is only valid in the context of the eth-device.

Now, if "ifconfig" shows "inet6 fe80::4687:fcff:fe9e:4ac7 prefixlen 64 scopeid 0x20<link>"
then "fe80::4687:fcff:fe9e:4ac7" is NOT a connectable address, and syscall connect() typically fails with EINVAL.

To make it a valid address, it needs to be suffixed with a "%" and the device name, like:
fe80::4687:fcff:fe9e:4ac7%enp4s0

Either the resolver can return the link name attached to the address separated with a "%" char, or it needs to ignore link-local inet6 addresses.

[Andreas L (avl42)]

Afternotes: I only noticed the bug shortly after upgrading from Ubuntu 16.04 to Ubuntu 18.04

Seems like 16.04 didn't yet use systemd-resolved, but dnsmasq, which only returned ipv4 addresses.

[Andreas L (avl42)]

Most of the times, the first hit (namely the ipv4 address) is all that is used from the DNS query.

In my case, it is essentially a testcase for Tcl's socket, which tries to establish a connection to an unlistened port, and expects a "connection refused" error. But Tcl in this case(namely that the connection gets refused) attempts the connect on the next address-result, which is then the bad ipv6-address. That one then caused an "EINVAL" from the connect() syscall, which it reported up, so the testcase expecting "connection refused" failed.

I'm aware that the impact is not very big, but that just explains, why it wasn't already noticed long before.

Conclusion is, that DNS should either not return an inet6 link-local address at all, as this cannot be used that way (namely without the name of the network-device), or it should only provide it together with the network-device, if that is possible in the DNS protocol.

[Dan Streetman (ddstreet)]

I can't reproduce this, can you provide specific cmdline steps to reproduce please.

[Andreas L (avl42)]

The premis is that you have a network device with a "link local" (aka: "scopeid 0x20<link>") inet6 address.

In my case, "ifconfig enp4s0" shows a line like this:
   inet6 fe80::4687:fcff:fe9e:4ac7 prefixlen 64 scopeid 0x20<link>

If your machine does NOT have such a line, then I must admit, I don't know how to produce one. Mine had it from default setup.

If your machine DOES have an inet6 link local address, then (in a shell) run "host $(hostname)" and it is likely to return an inet (ipv4) address (that's just fine) AND the link local inet6 address.

If your "host $(hostname)" doesn't give you any inet6 address, or gives you one with %network-name appended, then you might have a different version of "systemd" than 237-3ubuntu10.33 (as included in 18.04)

[Andreas L (avl42)]

If you see the bare inet6 link local address, but just can't see a problem with it,
then try:
$ telnet $(hostname) # some port that isn't listened on.
Trying 192.168.0.1...
Trying fe80::4687:fcff:fe9e:4ac7...
telnet: Unable to connect to remote host: Invalid argument

The symptom being here that you get "Invalid argument" instead of "Connection refused".

[Dan Streetman (ddstreet)]

> In my case, "ifconfig enp4s0" shows a line like this:
> inet6 fe80::4687:fcff:fe9e:4ac7 prefixlen 64 scopeid 0x20<link>

this is normal, when ipv6 is enabled; all interfaces get a link-local address.

> run "host $(hostname)" and it is likely to return...the link local inet6 address.

Your assumption that systemd-resolved will perform $(hostname) resolution to the system's ipv6 link local addresses is wrong; it doesn't do that, unless you've configured things that way. If your system does do that, something else is going on. Can you provide a *specific* example of what you are seeing on your system, instead of trying to generalize your problem.

[Andreas L (avl42)]

It was no "assumption", but an "observation".

$ nslookup $(hostname) # a blank name without any domain
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: xxx # my "$(hostname)"
Address: 192.168.0.1
Name: xxx
Address: fe80::4687:fcff:fe9e:4ac7

$ sudo fuser -n udp 53
53/udp: 22620
$ ps uww 22620
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
systemd+ 22620 0.0 0.0 52368 2220 ? Ss Nov29 1:50 /lib/systemd/systemd-resolved
$ dpkg -S /lib/systemd/systemd-resolved
systemd: /lib/systemd/systemd-resolved
$ dpkg -l systemd | cat # (no UUOC, but prevent dpkg from fitting output to terminal)
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-================-============-=================================
ii systemd 237-3ubuntu10.33 amd64 system and service manager
$

At which points does your output differ?

[Andreas L (avl42)]

If your (same-version?) systemd does not return the link local address to nslookup, then I'd appreciate a hint for what kind of config would make it do so, so I could check the particular files.

I just recently upgraded my machine from 16.04 (with dnsmasq) to 18.04 -- if I had changed any systemd-config I'd pretty surely remember. If systemd uses some of dnsmasq's config, there is a chance of customizations so long ago, that I might have forgotten about them.

[Andreas L (avl42)]

PS: re "assumption"/"observation" -- ok, that it would happen for you was indeed just an assumption.

[Dan Streetman (ddstreet)]

> It was no "assumption", but an "observation".

no, you are assuming that systemd-resolved is the authoritative nameserver providing the $(hostname) addresses, which is wrong. systemd-resolved is a stub nameserver that does no authoritative resolution itself; it gets its answers from other nameservers.

> Name: xxx # my "$(hostname)"

why are you obfuscating your hostname? It could be relevant, for example if you're choosing to use the 'local' domain.

Let's try this instead; do this:

$ sudo systemctl --runtime edit systemd-resolved

in the editor that opens, put this:

[Service]
Environment=SYSTEMD_LOG_LEVEL=debug

then, save it and exit the editor. Then do:

$ sudo systemctl restart systemd-resolved

and now lookup your name:

$ nslookup $(hostname)

and then do this and paste/attach the results:

$ journalctl -b -u systemd-resolved --no-pager

[Andreas L (avl42)]

Well, ok, here is the real data: my hostname is "fpc" and really has IPv4 10.2.2.1, the next upstream nameserver is another machine in my home network, 10.2.2.3 which is configured to also use its /etc/hosts (so I only need to maintain it on one machine). Doing the nslookup directly on 10.2.2.3 ONLY gives the ipv4 address (as I had already checked before opening this ticket, and which also made me very positive that the local resolver was to blame).

Unfortunately, the log doesn't really show much about where it gets what data from...

Dez 06 09:02:14 fpc systemd-resolved[25637]: Got DNS stub UDP query packet for id 24958
Dez 06 09:02:14 fpc systemd-resolved[25637]: Looking up RR for fpc IN A.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Sending response packet with id 24958 on interface 1/AF_INET.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Processing query...
Dez 06 09:02:14 fpc systemd-resolved[25637]: Got DNS stub UDP query packet for id 63588
Dez 06 09:02:14 fpc systemd-resolved[25637]: Looking up RR for fpc IN AAAA.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Sending response packet with id 63588 on interface 1/AF_INET.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Processing query...
Dez 06 09:02:14 fpc systemd-resolved[25637]: Got DNS stub UDP query packet for id 13823
Dez 06 09:02:14 fpc systemd-resolved[25637]: Looking up RR for fpc IN MX.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Cache miss for fpc IN MX
Dez 06 09:02:14 fpc systemd-resolved[25637]: Transaction 22387 for <fpc IN MX> scope dns on */*.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Using feature level UDP for transaction 22387.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Using DNS server 10.2.2.3 for transaction 22387.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Sending query packet with id 22387.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Processing query...
Dez 06 09:02:14 fpc systemd-resolved[25637]: Processing incoming packet on transaction 22387. (rcode=SUCCESS)
Dez 06 09:02:14 fpc systemd-resolved[25637]: Not caching negative entry without a SOA record: fpc IN MX
Dez 06 09:02:14 fpc systemd-resolved[25637]: Transaction 22387 for <fpc IN MX> on scope dns on */* now complete with <success> from network (unsigned).
Dez 06 09:02:14 fpc systemd-resolved[25637]: Sending response packet with id 13823 on interface 1/AF_INET.
Dez 06 09:02:14 fpc systemd-resolved[25637]: Freeing transaction 22387.

At that minute&second I did a "host fpc". The log entries before and after had sufficiently separate timings. (I repeated it a few times, until other requests from other applications didn't mix in.)

This is what I get from asking "upstream" DNS directly:
$ nslookup fpc 10.2.2.3
Server: 10.2.2.3
Address: 10.2.2.3#53

Name: fpc
Address: 10.2.2.1

$
That machine 10.2.2.3 is still running Ubuntu 16.04 and dnsmasq.

[Andreas L (avl42)]

It occurred to me, that the actual query might be cached at that time, so I did a fresh restart of systemd-resolved and query, and then follows the log for these times: I don't see much relevant details in it, except that it also tries to find MX-records but fails till <attempts-max-reached>.

avl@fpc:~$ date; sudo systemctl restart systemd-resolved
Fr Dez 6 09:27:34 CET 2019
avl@fpc:~$ date; host fpc
Fr Dez 6 09:27:39 CET 2019
fpc has address 10.2.2.1
fpc has address 10.2.2.250
fpc has IPv6 address fe80::4687:fcff:fe9e:4ac7
avl@fpc:~$

Note: the 10.2.2.250 is a remainder from a recent experiment, adding a second IP address to the same network device. It is not in 10.2.2.3's /etc/hosts . The problem of bad inet6 returned predates my addition of the second ip-address.

Here the log:

Dez 06 09:27:34 fpc systemd[1]: Stopping Network Name Resolution...
Dez 06 09:27:34 fpc systemd-resolved[4579]: Removing scope on link *, protocol dns, family *
Dez 06 09:27:34 fpc systemd[1]: Stopped Network Name Resolution.
Dez 06 09:27:34 fpc systemd[1]: Starting Network Name Resolution...
Dez 06 09:27:35 fpc systemd-resolved[4616]: Positive Trust Anchors:
Dez 06 09:27:35 fpc systemd-resolved[4616]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Dez 06 09:27:35 fpc systemd-resolved[4616]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dez 06 09:27:35 fpc systemd-resolved[4616]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Dez 06 09:27:35 fpc systemd-resolved[4616]: Using system hostname 'fpc'.
Dez 06 09:27:35 fpc systemd-resolved[4616]: New scope on link *, protocol dns, family *
Dez 06 09:27:35 fpc systemd-resolved[4616]: Found new link 2/enp4s0
Dez 06 09:27:35 fpc systemd-resolved[4616]: Found new link 1/lo
Dez 06 09:27:35 fpc systemd-resolved[4616]: Bus n/a: changing state UNSET → OPENING
Dez 06 09:27:35 fpc systemd-resolved[4616]: Bus n/a: changing state OPENING → AUTHENTICATING
Dez 06 09:27:35 fpc systemd-resolved[4616]: Creating stub listener using UDP/TCP.
Dez 06 09:27:35 fpc systemd-resolved[4616]: Switching to system DNS server 10.2.2.3.
Dez 06 09:27:35 fpc systemd-resolved[4616]: Bus n/a: changing state AUTHENTICATING → HELLO
Dez 06 09:27:35 fpc systemd-resolved[4616]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Dez 06 09:27:35 fpc systemd-resolved[4616]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 signature=su error-name=n/a error-message=n/a
Dez 06 09:27:35 fpc systemd-resolved...

[Andreas L (avl42)]

Here is some more info, namely an "strace" of the systemd-resolved program.

(An "lsof"-excerpt with the open channels follows the trace-log, to make sense of filedescriptors)

In a nutshell: at some point, it receives the inet6 address "fe80::4687:fcff:fe9e:4ac7" in an "recvmsg" call on FD 9, which lsof prints as:
"systemd-r 4966 systemd-resolve 9u netlink 0t0 15693998 ROUTE"
I don't understand what lsof tries to tell me here, but naively I'd assume that systemd-resolved also queries the routing table, and for that I also attach the output of "route -n -6" at the end of this message.

strace: Process 4966 attached
epoll_wait(4, [{EPOLLIN, {u32=3769468528, u64=94647668803184}}], 14, -1) = 1
clock_gettime(CLOCK_BOOTTIME, {tv_sec=1269407, tv_nsec=930758302}) = 0
recvfrom(12, NULL, 0, MSG_PEEK|MSG_TRUNC, NULL, NULL) = 21
recvmsg(12, {msg_name={sa_family=AF_INET, sin_port=htons(55365), sin_addr=inet_addr("127.0.0.1")}, msg_namelen=128->16, msg_iov=[{iov_base="j\204\1\0\0\1\0\0\0\0\0\0\3fpc\0\0\1\0\1", iov_len=3928}], msg_iovlen=1, msg_control=[{cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=IP_PKTINFO, cmsg_data={ipi_ifindex=if_nametoindex("lo"), ipi_spec_dst=inet_addr("127.0.0.53"), ipi_addr=inet_addr("127.0.0.53")}}, {cmsg_len=20, cmsg_level=SOL_IP, cmsg_type=IP_TTL, cmsg_data=[64]}], msg_controllen=56, msg_flags=0}, 0) = 21
stat("/etc/hosts", {st_mode=S_IFREG|0644, st_size=356, ...}) = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=324, ...}) = 0
clock_gettime(CLOCK_BOOTTIME, {tv_sec=1269407, tv_nsec=931200245}) = 0
sendto(9, {{len=24, type=RTM_GETADDR, flags=NLM_F_REQUEST|NLM_F_ACK|NLM_F_DUMP, seq=9, pid=0}, {ifa_family=AF_INET, ifa_prefixlen=32, ifa_flags=0, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=0}}, 24, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 16) = 24
recvmsg(9, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=128->12, msg_iov=[{iov_base=NULL, iov_len=0}], msg_iovlen=1, msg_control=[{cmsg_len=20, cmsg_level=SOL_NETLINK, cmsg_type=0x3}], msg_controllen=24, msg_flags=MSG_TRUNC}, MSG_PEEK|MSG_TRUNC) = 256
recvmsg(9, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=128->12, msg_iov=[{iov_base=[{{len=76, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=9, pid=4966}, {ifa_family=AF_INET, ifa_prefixlen=8, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_HOST, ifa_index=if_nametoindex("lo")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 127.0.0.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 127.0.0.1}, {{nla_len=7, nla_type=IFA_LABEL}, "lo"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHEINFO}, {ifa_prefered=4294967295, ifa_valid=4294967295, cstamp=3449, tstamp=3449}}]}, {{len=88, type=RTM_NEWADDR, flags=NLM_F_MULTI, seq=9, pid=4966}, {ifa_family=AF_INET, ifa_prefixlen=24, ifa_flags=IFA_F_PERMANENT, ifa_scope=RT_SCOPE_UNIVERSE, ifa_index=if_nametoindex("enp4s0")}, [{{nla_len=8, nla_type=IFA_ADDRESS}, 10.2.2.1}, {{nla_len=8, nla_type=IFA_LOCAL}, 10.2.2.1}, {{nla_len=8, nla_type=IFA_BROADCAST}, 10.2.2.255}, {{nla_len=11, nla_type=IFA_LABEL}, "enp4s0"}, {{nla_len=8, nla_type=IFA_FLAGS}, IFA_F_PERMANENT}, {{nla_len=20, nla_type=IFA_CACHE...

[Dan Streetman (ddstreet)]

> I don't understand what lsof tries to tell me here, but naively I'd assume that systemd-resolved also queries the routing table, and for that I also attach the output of "route -n -6" at the end of this message.

that's not correct.

What's the contents of your /etc/hosts file? Contents of /etc/resolv.conf?

What do these show:

$ systemd-resolve --status

$ systemd-resolve fpc

$ systemd-resolve -t A fpc

$ systemd-resolve -t AAAA fpc

$ dig @10.2.2.3 -t A fpc

$ dig @10.2.2.3 -t AAAA fpc

$ dig @127.0.0.53 -t A fpc

$ dig @127.0.0.53 -t AAAA fpc

[Andreas L (avl42)]

/etc/hosts:

127.0.0.1 localhost
127.0.1.1 kato i7

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

$ systemd-resolve --status
Global
         DNS Servers: 10.2.2.3
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (enp4s0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Remark: I have no idea where these ##.172.in-addr.arpa come from.

$ systemd-resolve fpc
fpc: 10.2.2.1%enp4s0
     10.2.2.250%enp4s0
     fe80::4687:fcff:fe9e:4ac7%2%enp4s0

-- Information acquired via protocol DNS in 1.6ms.
-- Data is authenticated: yes

Remark: ah, here it still has the network-name along with the ipv6

$ systemd-resolve -t A fpc
fpc IN A 10.2.2.1 # interface enp4s0
fpc IN A 10.2.2.250 # interface enp4s0

-- Information acquired via protocol DNS in 2.2ms.
-- Data is authenticated: yes

$ systemd-resolve -t AAAA fpc
fpc IN AAAA fe80::4687:fcff:fe9e:4ac7 # interface enp4s0

-- Information acquired via protocol DNS in 2.1ms.
-- Data is authenticated: yes

Remark: ah, here it also has the network-name along with the ipv6

$ dig @10.2.2.3 -t A fpc

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> @10.2.2.3 -t A fpc
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57577
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1280
;; QUESTION SECTION:
;fpc. IN A

;; ANSWER SECTION:
fpc. 0 IN A 10.2.2.1

;; Query time: 0 msec
;; SERVER: 10.2.2.3#53(10.2.2.3)
;; WHEN: Fri Dec 06 17:11:19 CET 2019
;; MSG SIZE rcvd: 48

Remark: As I said, upstream DNS 10.2.2.3 doesn't know about the second 10.2.2.250 address.

$ dig @10.2.2.3 -t AAAA fpc

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> @10.2.2.3 -t AAAA fpc
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 924
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;fpc. IN AAAA

;; Query time: 0 msec
;; SERVER: 10.2.2.3#53(10.2.2....

[Dan Streetman (ddstreet)]

You missed contents of /etc/resolv.conf

Also paste contents of /etc/systemd/resolved.conf

[Andreas L (avl42)]

Apologies...

I'll leave out the comment-headers and just paste the relevant contents:

/etc/resolv.conf:
nameserver 127.0.0.53
nameserver 10.2.2.3

/etc/systemd/resolved.conf:
[Resolve]
#DNS=
#FallbackDNS=
#Domains=
#LLMNR=no
#MulticastDNS=no
#DNSSEC=no
#Cache=yes
#DNSStubListener=yes

[Dan Streetman (ddstreet)]

> /etc/resolv.conf:
> nameserver 127.0.0.53
> nameserver 10.2.2.3

You shouldn't hand-edit your resolv.conf file; don't put the upstream nameserver in there.

Anyway, assuming you haven't edited out the search domain because you thought it wasn't relevant, you're using single-label hostnames, which shouldn't be used. resolved refuses to perform upstream lookups for single-label domains, and you don't have your hostname defined in your /etc/hosts file, so resolved has nobody to ask to lookup your hostname.

You should set up your network using a dns domain.

For reference:
https://www.iab.org/documents/correspondence-reports-documents/2013-2/iab-statement-dotless-domains-considered-harmful/

[Andreas L (avl42)]

Ok, just to check, I changed /etc/resolv.conf to remove second
nameserver, added "domain .hm" and "search hm", added "fpc.hm"
to the respective entry in 10.2.2.3's /etc/hosts.

Result is, that localhost's systemd-resolved now no longer
knows about upstream dns 10.2.2.3 -- where else am I supposed
to configure that?

So, I re-added "nameserver 10.2.2.3" to /etc/resolv.conf, and
it really seems like it would once again take the second
nameserver-entry as that for upstream-dns.

Apparently, with "domain" and "search" in place, systemd-resolved
no longer makes any attempt to obtain information from routing
table, thus appears to work.

Nevertheless, apparently systemd-resolved DOES have a feature
where it queries local information about network devices to
get a list of ip-addresses whenever there is no domain/search
defined and a bare name queried. What is the reason behind
that feature, if the suggested solution to this ticket is just:
"don't use bare hostnames"?

[Andreas L (avl42)]

I do agree that it is correct(TM) behaviour to not ask upstream DNS for bare ("single label") names. But this is not what this bugreport was about.

The point is what systemd-resolved does in such a case, where it cannot consult upstream DNS, and also doesn't find the name in local /etc/hosts:

It consults the routing table, and, when doing so, it deals wrongly with link-local inet6 addresses. That's what this bugreport was about.

[Dan Streetman (ddstreet)]

> Result is, that localhost's systemd-resolved now no longer
> knows about upstream dns 10.2.2.3 -- where else am I supposed
> to configure that?

If you use systemd-networkd to configure your network (or, netplan, which will indirectly configure networkd or network-manager for you), then systemd-resolved will be updated to know what your actual upstream nameserver(s) are; you can query what current upstream nameservers resolved is using with the 'systemd-resolve --status' command.

If you use ifupdown, there is a dhclient hook that tells resolved (or resolvconf) about your upstream nameservers.

If you manually configure your network, it's up to you to also manually configure your /etc/resolv.conf file, or manually tell resolved about your upstream nameservers, which you can do with the systemd-resolved command (see its man page).

> It consults the routing table, and, when doing so, it deals wrongly with link-local
> inet6 addresses. That's what this bugreport was about.

Why is it wrong? All the ipv6 addresses it resolves for its own hostname are reachable locally.

[Andreas L (avl42)]

> If you manually configure your network, ...

Ok, so adding it to /etc/resolv.conf is actually one of the supported ways.
Thanks (for info). I'll stick to that.

> Why is it wrong?

try connecting to that address:

avl@fpc:~$ telnet fe80::4687:fcff:fe9e:4ac7 22
Trying fe80::4687:fcff:fe9e:4ac7...
telnet: Unable to connect to remote host: Invalid argument

The link local inet6 address alone is useless information,...

avl@fpc:~$ telnet fe80::4687:fcff:fe9e:4ac7%enp4s0 22
Trying fe80::4687:fcff:fe9e:4ac7%enp4s0...
Connected to fe80::4687:fcff:fe9e:4ac7%enp4s0.
Escape character is '^]'.
SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3

... unless the network device name (enp4s0 on my machine) is
attached to the inet6 address.

[Andreas L (avl42)]

PS: I also appreciate all the other mechanisms for having upstream dns set up from dhcp or other dynamic connections. Just that one machine is set up statically.

[Andreas L (avl42)]

Is the factual incorrectness of "All the ipv6 addresses it resolves for its own hostname are reachable locally" understood? They are *only* reachable, if the interface is provided along with the link local inet6-address, but that extra condition isn't met.

[Dan Streetman (ddstreet)]

> Ok, so adding it to /etc/resolv.conf is actually one of the supported ways.

well, I wouldn't call it 'supported' as you're then responsible for managing the resolv.conf file, but it will bypass resolved completely (assuming you don't put 127.0.0.53 in the file, and you only use your upstream nameserver(s)).

The normal glibc getaddrinfo() that uses the resolv.conf entries to find nameserver(s) to query is of course supported no matter what you have in the resolv.conf file, but that rarely has bugs.

> try connecting to that address:
>
> avl@fpc:~$ telnet fe80::4687:fcff:fe9e:4ac7 22
> Trying fe80::4687:fcff:fe9e:4ac7...
> telnet: Unable to connect to remote host: Invalid argument

you want to telnet to your local host?

anyway, Ubuntu isn't the right place to take this up - you should open an upstream bug to systemd if this is an issue for you.

[Andreas L (avl42)]

> but it will bypass resolved completely

In my resolv.conf I have now both 127.0.0.53 and the upstream dns each in their own "nameserver"-line. Just as in comment #17, but in the meantime I added domain and search lines.

> you want to telnet to your local host?

I really wanted to demonstrate what happens if I attempt to connect to a bare link local inet6 address as returned by systemd-resolved in the situation where it falls back to resolving $(hostname) using info from local network devices. "telnet" is just a simple tool to connect to tcp/ip endpoints, in case you didn't know, a bit like netcat, but more generally available ;-)

> anyway, Ubuntu isn't the right place to take this up

Good to know so early in the thread...

[Dan Streetman (ddstreet)]

> In my resolv.conf I have now both 127.0.0.53 and the upstream dns each in their own "nameserver"-line.

if you haven't told resolved about any upstream nameservers, there is no point in pointing resolv.conf to it.

> "telnet" is just a simple tool to connect to tcp/ip endpoints, in case you didn't know, a bit like netcat, but more generally available ;-)

lol, yes we're all aware of what telnet is :)

> Good to know so early in the thread...

If you do open an upstream bug, I'll be happy to follow it and backport any fix if appropriate, but I really do suggest you just fix your network to use a domain name.

[Andreas L (avl42)]

> you haven't told resolved about any upstream nameservers ...

As I said, my original resolv.conf just had two nameserver entries, the local one (127.0.0.53) and the upstream one.

From that, all the diagnose-calls we did during this thread seemed to show that systemd-resolved DID know the upstream DNS. It really looks like as if systemd-resolved looked for the configured nameservers, recognized itself and took the other one as upstream. See for example comment #15: "DNS Servers: 10.2.2.3" in "Global" section.

Will look for where to create ticket for upstream systemd tomorrow. or when I next have time for it...

[Dan Streetman (ddstreet)]

> It really looks like as if systemd-resolved looked for the configured nameservers, recognized itself and took the other one as upstream.

it may, it depends entirely on your network configuration, which I don't know.

> Will look for where to create ticket for upstream systemd tomorrow. or when I next have time for it...

https://github.com/systemd/systemd/issues

if you do open a bug, please note it here; it's quite possible upstream will agree and fix it.

I'll switch this back to 'confirmed' while waiting to hear an opinion from upstream.

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]