Search list in resolv.conf breaks resolving for that domain

[Pär Lindfors]

Ubuntu 17.04
systemd 232-21ubuntu5

Adding a domain to the search list in /etc/resolv.conf breaks resolving for that domain. Not only does the search list not get used as expected, but host names in the domain cannot be resolved by systemd-resolved at all.

I just ran into this after upgrading from ubuntu 16.04 to 17.04 which enabled systemd-resolved. I have for a long time used resolveconf to add a 'search my-domain'-line to my /etc/resolv.conf.

Example of expected behaviour. With Googles DNS server (8.8.8.8) and ubuntu.com in the search list in /etc/resolv.conf. Both dig and systemd-resolve can resolve www.ubuntu.com and www:

    $ cat /etc/resolv.conf
    nameserver 8.8.8.8
    search ubuntu.com
    $ dig +nostat +nocmd www.ubuntu.com
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55037
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;www.ubuntu.com. IN A

    ;; ANSWER SECTION:
    www.ubuntu.com. 501 IN A 91.189.89.115

    $ dig +search +nostat +nocmd www
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25772
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 512
    ;; QUESTION SECTION:
    ;www.ubuntu.com. IN A

    ;; ANSWER SECTION:
    www.ubuntu.com. 382 IN A 91.189.89.103

    $ systemd-resolve www.ubuntu.com
    www.ubuntu.com: 91.189.89.115

    -- Information acquired via protocol DNS in 2.7ms.
    -- Data is authenticated: no
    $ systemd-resolve www
    www: 91.189.90.59
         (www.ubuntu.com)

    -- Information acquired via protocol DNS in 3.8ms.
    -- Data is authenticated: no

Ubuntu 17.04 default config, with the systemd-resolved name server in /etc/resolv.conf and no search list. www.ubuntu.com can still be resolved correctly:

    $ cat /etc/resolv.conf
    nameserver 127.0.0.53
    $ dig +nostat +nocmd www.ubuntu.com
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 64646
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 65494
    ;; QUESTION SECTION:
    ;www.ubuntu.com. IN A

    ;; ANSWER SECTION:
    www.ubuntu.com. 482 IN A 91.189.89.110
    $ systemd-resolve www.ubuntu.com
    www.ubuntu.com: 91.189.90.58

    -- Information acquired via protocol DNS in 18.2ms.
    -- Data is authenticated: no

Broken behaviour, using the systemd-resolved name server and specify ubuntu.com in search list. Resolving fails for www.ubuntu.com and www, both using dig (DNS) and using sytemd-resolve:

    $ cat /etc/resolv.conf
    nameserver 127.0.0.53
    search ubuntu.com
    $ dig +nostat +nocmd www.ubuntu.com
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 33334
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 65494
    ;; QUESTION SECTION:
    ;www.ubuntu.com. IN A

    $ dig +search +nostat +nocmd www
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 50588
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

    ;; OPT PSEUDOSECTION:
    ; EDNS: version: 0, flags:; udp: 65494
    ;; QUESTION SECTION:
    ;www.ubuntu.com. IN A

    $ systemd-resolve www.ubuntu.com
    www.ubuntu.com: resolve call failed: No appropriate name servers or networks for name found
    $ systemd-resolve www
    www: resolve call failed: All attempts to contact name servers or networks failed

[Dimitri John Ledkov (xnox)]

In 17.10 this has changed to a behaviour consistent with e.g. 16.04.
Search domains are properly trusted and are configured in resolved via its dbus api.
The generated /etc/resolv.conf contains resolved stub resolver and search domains out of the box (those e.g. acquired via dhcp, or configured in /etc/systemd/resolved.conf) and things work as expected via either dig or NSS-based resolution.

Could you please try e.g. artful daily CD to see if things work right for you there?

[Matthias Fratz (matthias.fratz)]

I can confirm that the underlying bug still exists in today's artful nightly CD.

I wasn't able to get a search path to stick in /etc/resolv.conf, but setting Domains=ubuntu.com in /etc/systemd/resolved.conf triggers the same bug: It breaks resolution for ubuntu.com and www.ubuntu.com (and probably all other subdomains as well), and www doesn't resolve either. nslookup reports SERVFAIL; systemd-resolve reports "No appropriate name servers or networks for name found".

Testing this on my 17.04, I enabled systemd-resolved logging. The output for "affected" domains (ie. ones named in the search path) listed "LLMNR"; for other domains it listed "LLMNR" and "DNS" strategies. I will try to reproduce this in the artful VM and will hopefully be able to post the debugging output tomorrow.

[Dimitri John Ledkov (xnox)]

Hm.

So my local WiFi router is setup to send 'surgut.co.uk' domain. And I can resolve 'blog.surgut.co.uk' and 'surgut.co.uk' via both `systemd-resolve [blog.]surgut.so.uk` and `getent ahosts [blog.]surgut.co.uk`.

I think i really need to see all of the settings for systemd can you attach a tarball of:

/run/systemd/*
/etc/systemd/resolved.conf*

?

/me really should extend the apport hooks to include those things.

[Matthias Fratz (matthias.fratz)]

Our DHCP server delivers a search domain (inf.uni-konstanz.de) as well. This isn't enough to trigger the bug for me, though, at least on 17.04. (systemd-resolve doesn't actually USE the search path, so merkur236.inf.uni-konstanz.de works but merkur236 doesn't, but that's a different problem.)

What does trigger the bug on 17.04 is manually configuring a search path, like it seems the original reporter did:

$ cat /etc/resolvconf/resolv.conf.d/base
search disy.inf.uni-konstanz.de inf.uni-konstanz.de uni-konstanz.de

At some point this gets copied to /etc/resolv.conf. After a "service systemd-resolved restart", resolution is broken for those domains until I remove the search path and restart systemd-resolved again.

I couldn't yet reproduce this on 17.10 because restarting systemd-resolved overwrites /etc/resolv.conf. Where is the right place to configure a search path on 17.10? I don't see it anywhere in the GUI and I'm not sure whether /etc/systemd/resolved.conf is the right place.

I attached the files you suggested from a fresh 17.10 install, with "Domains=ubuntu.com" manually added to /etc/systemd/resolved.conf. Adding that line (and restarting systemd-resolved) breaks resolving; removing (and restarting) restores functionality.

[Dimitri John Ledkov (xnox)]

On 10 October 2017 at 15:39, Matthias Fratz <email address hidden> wrote:
> Our DHCP server delivers a search domain (inf.uni-konstanz.de) as well.
> This isn't enough to trigger the bug for me, though, at least on 17.04.
> (systemd-resolve doesn't actually USE the search path, so merkur236.inf
> .uni-konstanz.de works but merkur236 doesn't, but that's a different
> problem.)
>
> What does trigger the bug on 17.04 is manually configuring a search
> path, like it seems the original reporter did:
>
> $ cat /etc/resolvconf/resolv.conf.d/base
> search disy.inf.uni-konstanz.de inf.uni-konstanz.de uni-konstanz.de
>
> At some point this gets copied to /etc/resolv.conf. After a "service
> systemd-resolved restart", resolution is broken for those domains until
> I remove the search path and restart systemd-resolved again.
>
> I couldn't yet reproduce this on 17.10 because restarting systemd-
> resolved overwrites /etc/resolv.conf. Where is the right place to
> configure a search path on 17.10? I don't see it anywhere in the GUI and
> I'm not sure whether /etc/systemd/resolved.conf is the right place.
>

The really correct way on 17.10 is to stop using ifupdown and stop
using resolvconf and stop using libnss-resolve

$ upgrade to 17.10
$ sudo netplan ifupdown-migrate
$ sudo apt remove --purge ifupdown
$ sudo apt remove --purge resolvconf
$ sudo apt remove --purge libnss-resolve
$ sudo reboot

At this point, /etc/resolv.conf will start pointing at a resolved
managed stub-resolv.conf file which will automatically pick up and use
DHCP provided domains.
Further domains configuration can be provided by adjusting
/etc/netplan/*.yaml file, or by adjusting systemd-networkd drop-ins,
or by adjusting /etc/systemd/resolved.conf.
It's best for the search domains to be provided over DHCP instead of
hardcoding the config in files on disk.

You may also choose to use networks: backend: network-manager in the
netplan.yaml file, if you wish to use NetworkManager instead of the
networkd as the networking management backend.

Search domains specified in resolvconf syntax / directly in the
/etc/resolv.conf will not be read/used by resolved stub resolver, and
will not be used by libnss-resolve, thus if you wish to keep using
resolvconf/ifupdown one will need to provide the search domain to
resolved separately. Best way to do that would be via
/etc/systemd/resolved.conf.

--
Regards,

Dimitri.

[Matthias Fratz (matthias.fratz)]

Tried that, and it started using the DHCP-provided search path (yay!).

Setting the search path in NetworkManager (which is responsible for the interface in question) works, ie. honors the search path and doesn't break resolving for those domains, with both single and multiple search paths:

[ipv4]
dns-search=disy.inf.uni-konstanz.de;inf.uni-konstanz.de;uni-konstanz.de
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=disy.inf.uni-konstanz.de;inf.uni-konstanz.de;uni-konstanz.de
method=auto

Having to do this for each connection and for both IPv4 and IPv6 sucks, but it's better than not having a search path.

Trying to set the search path to Domains=ubuntu.com globally in resolved.conf still breaks ubuntu.com, of course. Out of curiosity, I then put this in resolved.conf:

Domains=uni-konstanz.de inf.uni-konstanz.de disy.inf.uni-konstanz.de ubuntu.com

This works for the domains listed in the interface, honoring the search path and correctly resolving both short (git) and long (git.uni-konstanz.de) domains. But it breaks resolution completely for ubuntu.com and subdomains.

So: Does systemd-resolved need to have a network interface "associated" with each search domain?? This is very much not how DNS works but it's a boundary case that might be easy to get wrong.

(This is all on the 17.10 VM, and with resolved.conf empty apart from [Resolve] and the Domains= line, where mentioned.)

[Dimitri John Ledkov (xnox)]

On 11 October 2017 at 15:25, Matthias Fratz <email address hidden> wrote:
> Tried that, and it started using the DHCP-provided search path (yay!).
>
> Setting the search path in NetworkManager (which is responsible for the
> interface in question) works, ie. honors the search path and doesn't
> break resolving for those domains, with both single and multiple search
> paths:
>
> [ipv4]
> dns-search=disy.inf.uni-konstanz.de;inf.uni-konstanz.de;uni-konstanz.de
> method=auto
>
> [ipv6]
> addr-gen-mode=stable-privacy
> dns-search=disy.inf.uni-konstanz.de;inf.uni-konstanz.de;uni-konstanz.de
> method=auto
>
> Having to do this for each connection and for both IPv4 and IPv6 sucks,
> but it's better than not having a search path.
>
>
> Trying to set the search path to Domains=ubuntu.com globally in resolved.conf still breaks ubuntu.com, of course. Out of curiosity, I then put this in resolved.conf:
>
> Domains=uni-konstanz.de inf.uni-konstanz.de disy.inf.uni-konstanz.de
> ubuntu.com
>
> This works for the domains listed in the interface, honoring the search
> path and correctly resolving both short (git) and long (git.uni-
> konstanz.de) domains. But it breaks resolution completely for ubuntu.com
> and subdomains.
>
> So: Does systemd-resolved need to have a network interface "associated"
> with each search domain?? This is very much not how DNS works but it's a
> boundary case that might be easy to get wrong.
>
> (This is all on the 17.10 VM, and with resolved.conf empty apart from
> [Resolve] and the Domains= line, where mentioned.)
>

If there is per-interface configuration available resolved will use
that, and it is preferred mode of operation. Anything else is
ambiguous.

This is to support split-dns situations such that company.internal.vpn
on a a VPN interface can have Domains specified and thus not leak
VPN-intended queries to the general intenet / gateway nameserver.

I'm still struggling to comprehend the obsession of adding
"ubuntu.com" in your examples. Please stop doing that. This is not a
domain you control, and not something one should be trying to
override, as that carries risk of failing to resolve or miss-resolve
domain names used for updates.

If DHCP is not providing you the correct domains all clients should be
using on a given connection -> please fix your DHCP server config.
If that is not possilbe -> you can fix that up locally on per-connection basis.
Lease ubuntu.com alone.

Can you describe in general terms, what network configuration exists,
and how is it broken by default when artful is used as a DHCP client?
It is intentional that DHCP server is not providing the correct search
domains? Why are you overriding them on each client? Why are you
trying to override resolution of ubuntu.com domains?

It is intentional that one has to maintain correct per-link
configuration. This used to happen with resolvconf, as each dhcp v4
and v6 configs were kept separately internally, and were correctly
removed each time a lease/link was lost. Now in addition to keeping
track of which nameserver belogs to which link, we also only send
queries to the right nameservers and matching domains by default. This
improves security and privacy...

[Matthias Fratz (matthias.fratz)]

As for ubuntu.com, that was only as an example to show a situations when things break. I really just copied what the original submitter used -- I should probably have used example.com instead though, that's less confusing.

My concrete situation is that the server delivers only inf.uni-konstanz.de because that's what matches the university's organizational structure. (And having it changed it is not an option.) I put the other two (disy.inf.uni-konstanz.de and uni-konstanz.de) in the search path long ago because I regularly connect to hosts in those domains. Saves a lot of typing errors, and AFAIK it's not an uncommon use case for the search path. But this broke DNS resolution for disy.inf.uni-konstanz.de and uni-konstanz.de (probably because they're not in any per-interface search path) on upgrade to 17.04, and got me to this bug.

As far as I can tell now, the bug should only affect users on or upgrading to zesty who have manually configured a search path using resolvconf, and artful users who configure one in resolved.conf. That's clearly a minority of users.

I certainly agree to the "only send queries to the right nameserver" part. That makes sense, as does UseDomains and setting it to true by default. What I find troubling is that it seems to just swallow the query when there's no interface matching a search path entry. That doesn't agree with the documentation of either Domains= or UseDomains=, which both suggest that one can have search paths not associated with an interface. But I guess that's a bug in the systemd-resolved and not in Ubuntu, so I'll probably just report it upstream.

[Dimitri John Ledkov (xnox)]

On 12 October 2017 at 14:55, Matthias Fratz <email address hidden> wrote:
> As for ubuntu.com, that was only as an example to show a situations when
> things break. I really just copied what the original submitter used -- I
> should probably have used example.com instead though, that's less
> confusing.
>
> My concrete situation is that the server delivers only inf.uni-
> konstanz.de because that's what matches the university's organizational
> structure. (And having it changed it is not an option.) I put the other
> two (disy.inf.uni-konstanz.de and uni-konstanz.de) in the search path
> long ago because I regularly connect to hosts in those domains. Saves a
> lot of typing errors, and AFAIK it's not an uncommon use case for the
> search path. But this broke DNS resolution for disy.inf.uni-konstanz.de
> and uni-konstanz.de (probably because they're not in any per-interface
> search path) on upgrade to 17.04, and got me to this bug.
>
> As far as I can tell now, the bug should only affect users on or
> upgrading to zesty who have manually configured a search path using
> resolvconf, and artful users who configure one in resolved.conf. That's
> clearly a minority of users.
>
> I certainly agree to the "only send queries to the right nameserver"
> part. That makes sense, as does UseDomains and setting it to true by
> default. What I find troubling is that it seems to just swallow the
> query when there's no interface matching a search path entry. That
> doesn't agree with the documentation of either Domains= or UseDomains=,
> which both suggest that one can have search paths not associated with an
> interface. But I guess that's a bug in the systemd-resolved and not in
> Ubuntu, so I'll probably just report it upstream.
>

I'd rather report the bug upstream myself, as upstream prefer
downstream distributions to curate these things.

Clearly, as an end-user you cannot fix DHCP server config if it
provides insufficient search paths.

So you are saying that you have a.example.com domain configured on the
link, with a nameserver W.X.Y.Z which can also resolve b.example.com
domains and you want it to do so, and when setting Domains in
resolved.conf it does not result in resolving b.example.com domains in
addition to a.example.com domains? I'm not sure if this is a bug or a
feature =))))))

The right answer is probably to have per-link configuration encoded in
netplan/networkd/networkmanager, but that might be annoying to do if
one connects to said network over wifi, over usb ethernet on different
usb ports, and multiple vlan end-points. As I guess on such a network
all of these connections are valid, and all of them will receive
incomplete search list and it would be convenient to set a global
list.

I'll play around with this.

--
Regards,

Dimitri.

[Matthias Fratz (matthias.fratz)]

In fact it's slightly simpler in that both a.example.com and b.example.com are public domains. (This is why I put them in the global config to begin with; these domains will resolve over any nameserver.) Thus it's not so much that queries for b.example.com don't go to W.X.Y.Z; it's that they don't go anywhere.

More precisely: By default, ie. without a Domains= line in resolved.conf, b.example.com resolves correctly: the query goes to some random nameserver, which can resolve it because it's a public domain. With Domains=b.example.com, b.example.com fails to resolve, and systemd-resolve reports "No appropriate name servers or networks for name found" without querying any server. (At least I cannot observe any DNS or LLMNR traffic on the network connection of my test VM.)

[ts (toto-23)]

I seem to have a similar (the same) problem. Ubuntu 17.10 - sleep and resume leaves the resolv.conf with a "search" directive that breaks name resolution for any name within the domains mentioned in the search directive.

Trying to resolve any name fails, as e.g.:

% systemd-resolve <name in search directive>
<name in search directive>: resolve call failed: No appropriate name servers or networks for name found

Deleting the search directive and restarting the networking service resolves things temporarily (until the next suspend/resume loop).

I've removed the request for search directives from the dhcpclient.conf, and this works fine the first start, but suspend/resume leads some service to add the search directive again, and name resolution breaks.

[Matthias Fratz (matthias.fratz)]

@toto-23, you may want to try putting the following in /etc/systemd/resolved.conf as a temporary workaround:

[Resolve]
Domains=

Note that there is nothing after the equals sign. According to the docs it will override anything systemd-resolved reads from /etc/resolv.conf with an empty list. This is the workaround that works for me on 17.04, so maybe it also works on 17.10. On my 17.10 test VM it at least doesn't seem to cause any problems.

Interestingly, per-connection search paths still work for me (the documentation is ambiguous about whether Domains= overrides them or not) on both 17.04 and 17.10. So setting "Domains=" by default might be enough to fix the bug? Not sure whether that has unintended side effects, though.

[Launchpad Janitor (janitor)]

[Expired for systemd (Ubuntu) because there has been no activity for 60 days.]