The symbolic link /etc/resolv.conf points to the wrong file by default

[Glenn J. Schworak]

When using nslookup for local machine names, the local DNS was being ignored (not queried) and none of the local machines could be found.

After much research and digging, it was discovered that the cause was the incorrect symbolic link /etc/resolv.conf file.

The default install caused systemd-resolve to configure the link to point to the stub file:
/etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

Reomving that link and pointing it to the correct file solved the DNS lookup issue. The correct link looks like this:
/etc/resolv.conf -> /run/systemd/resolve/resolv.conf

Steps used to test the bug before fixing the link is to perform an nslookup on a local (non FQDN) machine that is in your local DNS (my router is my DNS server for this case) Here is an example of the incorrect output:

$ nslookup web1

Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find web1: SERVFAIL

Switching the symbolic link solves the problem. Here is my solution:

$ sudo rm -f /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

After switching the symbolic link, the nslookup functions properly.

$ nslookup web1

Server: 192.168.1.1
Address: 192.168.1.1#53

Name: web1
Address: 192.168.1.107

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: systemd 237-3ubuntu10
ProcVersionSignature: Ubuntu 4.15.0-22.24-generic 4.15.17
Uname: Linux 4.15.0-22-generic x86_64
ApportVersion: 2.20.9-0ubuntu7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Fri Jun 1 05:28:41 2018
InstallationDate: Installed on 2018-01-20 (131 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Release amd64 (20180105.1)
MachineType: Dell Inc. Inspiron 5755
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.15.0-22-generic root=UUID=7fe151d3-4033-4903-b356-341d9f16e124 ro acpi=force
SourcePackage: systemd
UpgradeStatus: Upgraded to bionic on 2018-04-28 (33 days ago)
dmi.bios.date: 08/27/2015
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A08
dmi.board.name: 0VY15F
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 8
dmi.chassis.vendor: Dell Inc.
dmi.chassis.version: A08
dmi.modalias: dmi:bvnDellInc.:bvrA08:bd08/27/2015:svnDellInc.:pnInspiron5755:pvrA08:rvnDellInc.:rn0VY15F:rvrA00:cvnDellInc.:ct8:cvrA08:
dmi.product.name: Inspiron 5755
dmi.product.version: A08
dmi.sys.vendor: Dell Inc.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Ubfan (ubfan1)]

The link target, stub-resolv.conf is the correct file for enabling systemd-resolvd. Various problems with the systemd-resolvd may be fixed by installing the libnss-resolve package (bug 1769016). Blank local domain problems have their own bug, 1699660 (fix is to use ~.). This report probably should be marked "invalid".

[David Reagan (jerrac)]

I just ran into this issue myself on 20.04. Specifically I was adding my own internal nameserver to a vm.

@Ubfan, why is stub-resolv.conf the correct file?

The file that /etc/systemd/resolved.conf updates is /run/systemd/resolve/resolv.conf. As far as I can tell, the proper way to update /etc/resolv.conf is via /etc/systemd/resolved.conf. So if I want my changes to apply, then the /etc/resolv.conf link should point to the file systemd actually changes.

Or is there a different method of updating what nameservers your computer uses? If there is, why is systemd involved with resolv.conf at all?

Anyway, I believe this is a bug. Fortunately not one that effects people who don't know how to find the workarounds, but still a bug. I do hope it eventually gets fixed.

[Per Olav Kroka (perok)]

I just want to confirm that this is a problem. I have had it for a while and not found another workaround (until now) than restarting the machine.

If this is NOT a bug then something "close by" is a bug.

[Per Olav Kroka (per-olav-kroka)]

Please disregard my last comment (#5). The problem I had was not this one but related to using other connector agents in addition to NetworkManager, in my case the openvpn3 command line.

(Automatic connection to the VPN is not there, but that is probably safest.)

[Cody (qwertycody)]

I have been encountering this issue for the past month on my newly installed (1-2 months old) version of Ubuntu 20.04.2 LTS.

I finally got frustrated enough to dig in and find a solution and arrived here from a stackexchange comment.

FYI is all - OP's solution seems to have fixed the issue for me - will report back in a few days if not.

Why is this an issue on vanilla ubuntu?

Can you imagine how frustrated non-technical end users would be?

[Zhuokun Yang (jasdenty)]

OS: Ubuntu 18.04 bionic
Kernel: x86_64 Linux 4.19.17

For those who encounter situations otherwise, here is my case and workarounds.

my case:
---
 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

I remove the symlink and assign a new one
 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
reboot, and the /etc/resolv.conf says there is NO DNS SERVER. DNS Lookup is completeny unusable at this point.

If it points to the stub-resolv.conf variant, it seems that the systemd-resolved is not working properly, at 127.0.0.53.
---

my solution:
---
1. remove the symlink,
2. replace it with a solid file, either empty or with a line `nameserver 8.8.8.8`
3. reboot

and what you will see is a autogenerated resolv.conf made by NetworkManager, which changes dynaminally depending on what network you are connected to.

using netstat -tunlp to grep the nameserver in the file, you may see
```
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN 1604/dnsmasq
udp 0 0 127.0.1.1:53 0.0.0.0:* 1604/dnsmasq
```
which is also recommended in a lot of threads in askubuntu (i.e. google enable dns cache in ubuntu).
---

[aliengotithard (xmartinnovakx)]

UPDATE 2: In my case it's docker issue. I'm attaching link to docker draft with linked issues that ought to resolve this issue.
https://github.com/moby/moby/pull/41374

-------------------

UPDATE 1: after tinkering i'm not sure it's this bug im encountering as there are some old and new ways of configuring dns and i'm confused. So just take following comment lightly. Sorry.

It also might be following issue, but i can't recognise why it happened but i really think it was after "apt update".
https://github.com/docker/for-linux/issues/979

----------------------

ORIGINAL POST:

Also affects docker VPS - Is there a definitive solution? Im going to try @jasdenty solution. Im not that technical user.

It affects my VPS after "apt upgrade". Suddenly resolving domains from docker containers stopped - from what i understand docker defaultly inherits /etc/resolv.conf. Completly revisiting my docker networks didn't help.

So this comment is for you guys whose dockers stopped working after update.

Thanks

[Derek V Giroulle (d3r3klt)]

Update 2 : The experts at Ask Ubuntu don't seem to be able to come up with a solution this is a bug that blocks normal operation

Update 1 : This problem blocks normal usage of a laptop , access to Internet / VPN impossible

OS Ubuntu 20.04.3 LTS
After a recent update a working laptop suddently developped the following problem :
- DNS problems for al Internet related apps
- Ping : termporary failure in Name resolution
editing manually allows internet access for short term until systemd-resolve overwrites the file.
Its no use to replace /etc/resolv.conf with a file as it get overwritten regulary.

The stub-file /run/systemd/resolve/stub-resolv.conf contains

> nameserver 127.0.0.53
> options edsn0 trust-ad

In my Network options I have
DHCP auto
DNS manual 19*.2*8.*.21, 19*.2*8.*.22 ( which are my ISP dedicated DNS servers (stars are hiding numbers))

the result of command

    $ ls -al /etc/resolv.conf

    lrwxrwxrwx 1 root root 32 okt 13 14:10 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

when i overwrite /run/systemd/resolve/resolve.conf with
> nameserver 19*.2*8.*.21
> nameserver 19*.2*8.*.22
my internet connection temporarily works

The /run/systemd/resolve/resolve.conf contains
> nameserver ::1

I tried overwriting /run/systemd/resolve/resolv.conf with the correct nameservers and then applied your solution :

$ sudo rm -f /etc/resolv.conf
$ sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

And restarted the systemd-resolve service

and the /etc/resolv.conf and /run/systemd/resolve/resolv.conf files showed again

> nameserver ::1

i was asked output of resolvectl status

>Global
> LLMNR setting: no
> MulticastDNS setting: no
> DNSOverTLS setting: no
> DNSSEC setting: no
> DNSSEC supported: no
> DNSSEC NTA: 10.in-addr.arpa
> 16.172.in-addr.arpa
> 168.192.in-addr.arpa
> 17.172.in-addr.arpa
> 18.172.in-addr.arpa
> 19.172.in-addr.arpa
> 20.172.in-addr.arpa
> 21.172.in-addr.arpa
> 22.172.in-addr.arpa
> 23.172.in-addr.arpa
> 24.172.in-addr.arpa
> 25.172.in-addr.arpa
> 26.172.in-addr.arpa
> 27.172.in-addr.arpa
> 28.172.in-addr.arpa
> 29.172.in-addr.arpa
> 30.172.in-addr.arpa
> 31.172.in-addr.arpa
> corp
> d.f.ip6.arpa
> home
> internal
> intranet
> lan
> local
> private
> test

> Link 4 (ipv6leakintrf0)
> Current Scopes:
> DNS D...

[Guy Rouillier (guy-rouillier)]

I just encountered this issue upgrading Ubuntu MATE from 22.04 to 22.10. After the update, I had no network connectivity, and /etc/resolv.conf was symlinked to /run/systemd/resolve/stub-resolv.conf. I changed that to /run/systemd/resolve/resolve.conf, and my system is able to make connections once again. My system is running NetworkManager, and the only active connection is Docker0.

/run/systemd/resolve/stub-resolv.conf:

nameserver 127.0.0.53
options edns0 trust-ad
search DOMAINS

/run/systemd/resolve/resolv.conf

nameserver 192.168.2.1
nameserver 1.1.1.1
nameserver 1.0.0.1
search DOMAINS

[UPDATE]
After posting this, I found the following post about manually configuring DNS servers with systemd:

https://wiki.archlinux.org/title/systemd-resolved#Setting_DNS_servers

My Ubuntu MATE 22.10 does not have /etc/systemd/resolved.conf.d/dns_servers.conf, but it does have /etc/systemd/resolved.conf. So, I updated that with my DNS entries like so:

DNS=192.168.2.1 1.1.1.1 1.0.0.1

And I reset the /etc/resolv.conf symlink back to /run/systemd/resolve/stub-resolv.conf. Following a reboot, DNS is now working properly. There are a lot of moving pieces here, but it *appears* that to manually configure DNS servers, editing /etc/systemd/resolved.conf (or /etc/systemd/resolved.conf.d/dns_servers.conf if you have it) is the appropriate thing to do.

[Lukas Märdian (slyon)]

Ubuntu uses the local systemd-resolved DNS resolver by default.

/etc/resolv.conf being a symlink pointing to /run/systemd/resolve/stub-resolv.conf is the correct default setting. It will redirect any DNS request to sd-resolved's local resolver at 127.0.0.53.

This has the benefit of local caching and per-interface DNS configuration via sd-resolved.
Switching the symlink to /etc/resolv.conf -> /run/systemd/resolve/resolv.conf would make sd-resolved display the upstream DNS servers directly into the file, kind of a legacy/compatibility mode, loosing all the benefits of a local resolver. This is NOT recommended.

You can manually configure your DNS settings via netplan.io or sd-resolved directly, as stated in comment #13. NetworkManager should automatically detect sd-resolved and integrate nicely, some other (legacy) tools might not integrate as nicely.

If DNS is not working for you, you can configure a fallback, e.g. Google DNS:

$ cat /etc/systemd/resolved.conf.d/fallback_dns.conf
[Resolve]
FallbackDNS=8.8.8.8 2001:4860:4860::8888

I'm marking this bug as "Invalid". Please keep your symlink as /etc/resolv.conf -> /run/systemd/resolve/stub-resolv.conf

[Guy Rouillier (guy-rouillier)]

Lukas Märdian (slyon): "NetworkManager should automatically detect sd-resolved and integrate nicely"

Thank you for this guidance, Lukas. I had disabled NetworkManager on this system because it wasn't configuring my network connections as I needed them. I had moved my home LAN onto a separate subnet - 192.168.2.* - as my provider is Verizon FIOS, and it controls 192.168.1.*. Even though I put specific manual settings into NetworkManager to tell it to use 192.168.2.*, it was still picking DNS servers on 192.168.1.*.

But your post encouraged me to try again (2 years later) and this time I got NetworkManager working with Manual configuration. So, I now have everything configured as default with stub-resolv.conf and NetworkManager enabled, and it's all working correctly. The only place I have DNS servers specified is in NetworkManager, separately for each interface ("Wired connection 1" and "docker0"). I'm providing these details in case someone finds this report via search.