systemd-resolved breaks VPN with split-horizon DNS

Bug #1624317 reported by Anders Kaseorg on 2016-09-16
424
This bug affects 85 people
Affects Status Importance Assigned to Milestone
NetworkManager
Unknown
Unknown
network-manager (Ubuntu)
High
Unassigned
Zesty
High
Unassigned
Artful
High
Unassigned

Bug Description

[Impact]

 * NetworkManager incorrectly handles dns-priority of the VPN-like connections, which leads to leaking DNS queries outside of the VPN into the general internet.

 * Upstream has resolved this issue in master and 1.8 to correctly configure any dns backends with negative dns-priority settings.

[Test Case]

#FIXME#

 * detailed instructions how to reproduce the bug

 * these should allow someone who is not familiar with the affected
   package to reproduce the bug and verify that the updated package fixes
   the problem.

#FIXME#

[Regression Potential]

 * If this issue is changed DNS resolution will change, for certain queries, to go via VPN rather than general internet. And therefore, one may get new/different results or even loose access to resolve/access certain parts of the interent depending on what the DNS server on VPN chooses to respond to.

[Other Info]

 * Original bug report

I use a VPN configured with network-manager-openconnect-gnome in which a split-horizon DNS setup assigns different addresses to some names inside the remote network than the addresses seen for those names from outside the remote network. However, systemd-resolved often decides to ignore the VPN’s DNS servers and use the local network’s DNS servers to resolve names (whether in the remote domain or not), breaking the split-horizon DNS.

This related bug, reported by Lennart Poettering himself, was closed with the current Fedora release at the time reaching EOL:
https://bugzilla.redhat.com/show_bug.cgi?id=1151544

Martin Pitt (pitti) wrote :

This matches https://github.com/systemd/systemd/issues/3421 very closely (I've worked on this a bit, but it got stalled, sorry). I think that is the exact symptom you are seeing, can you confirm?

Changed in systemd (Ubuntu):
status: New → Incomplete
Anders Kaseorg (andersk) wrote :

3421 seems like the opposite of this problem. There, queries that belong on the local network instead go to the domain-limited VPN servers. Here, queries that should go to the VPN servers (which in my case are not domain-limited) instead go to the local servers. Also, here I am using NetworkManager, not systemd-networkd.

Changed in systemd:
status: Unknown → New
Anders Kaseorg (andersk) wrote :

(I believe the requested information has been provided.)

Changed in systemd (Ubuntu):
status: Incomplete → New
Martin Pitt (pitti) wrote :

The upstream issue actually applies to both. IMHO, if you restrict a DNS server to a particular list of domains it should be used *exactly* for the given domains (only). Querying it for other domains is a privacy leak, and querying other name servers for those domains is most probably going to fail anyway and thus a waste.

> Also, here I am using NetworkManager, not systemd-networkd.

That's unrelated, as that is about resolving DNS names, not bringing up the network.

So, this does match the upstream issue, setting to triaged.

Changed in systemd (Ubuntu):
status: New → Triaged
importance: Undecided → Medium
Anders Kaseorg (andersk) wrote :

I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

Anders Kaseorg (andersk) on 2016-09-29
tags: added: regression-release
tags: added: yakkety
Anders Kaseorg (andersk) on 2016-09-29
tags: removed: regression-release
Martin Pitt (pitti) wrote :

> I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

This is something different then; dropping the upstream bug link. So I'm afraid this needs a more precise description of how to set this up/reproduce this, and a debug output of resolved would be helpful:

  sudo systemctl stop sytemd-resolved
  sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved

then do the operation that fails, and copy&paste the output here. Thanks!

Changed in systemd:
importance: Unknown → Undecided
Changed in systemd (Ubuntu):
status: Triaged → Incomplete
Martin Pitt (pitti) wrote :

Note that in current Yakkety, NetworkManager uses dns=dnsmasq again, because until two weeks ago there was no resolved plugin for NetworkManager. That exists upstream now, but not yet in our packages, thus resolved only learned about these via dhcp-client writing the nameservers into /etc/resolv.conf and thus any association of name servers to their corresponding links was lost.

This will be fixed by the resolved plugin when NM can tell resolved about per-link DNS servers.

I think this might actually be what you mean with "split-horizon DNS". If it is, then this is fixed in yakkety, and will be fixed in z-series by using the resolved DNS plugin. If not, I don't know what you mean I'm afraid.

Ognjen (sekil) wrote :

Hello,

I'm having same issues with vpnc plugin and DNS servers I get from my VPN. Seems like dnsmasq doesn't try to send DNS resolve packet at all via tunnel. Once I connect to tunnel, DNS resolving is totally broken.

Ognjen (sekil) wrote :

This started on 16.10 btw.

stan383 (stan383) wrote :

The same problem here. Upgraded from 16.04 to 16.10. When using network manager for connecting to cisco VPN (network-manager-openconnect), DNS does not work at all. Even public domains are not resolved.
The only possibility for using VPN for me now is to manually add VPN's internal nameserver to /run/systemd/resolve/resolv.conf

Matthew General (mattgen88) wrote :

I am also affected. Broken both on a fresh install and an upgrade, dns for things behind the vpn do not resolve.

Dennis (deki24) wrote :

Same issue here after upgrade to 16.10.

Here is the log after connecting to VPN:
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.49.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.49 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.

Later on:
Lost too many UDP packets, downgrading feature level...
Using degraded feature set (UDP+EDNS0+DO) for DNS server 10.1.96.48.
Using feature level UDP+EDNS0+DO for transaction 35344.
Sending query packet with id 35344.
Timeout reached on transaction 29603.

Changed in systemd (Ubuntu):
status: Incomplete → Confirmed
mxCoder (ricardoe) wrote :

Hi,

I think I'm having the same issue:

Linux ricardo-N24-25BU 4.8.0-28-generic #30-Ubuntu SMP Fri Nov 11 14:03:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

Just upgraded to 16.10 last night, after bootup, I connect to a work VPN (OpenVPN + Pritunl) which was working ok so far.

route -n shows the new routes, and resolve.conf is correctly updated with the VPN DNS nameserver

Any direct query: dig, nslookup, ip route get {host} resolves correctly using the VPN DNS

But any other command: ping, telnet, mysql, etc resolves without the VPN DNS (unless instructed otherwise explicitely)

Quick solution was:

sudo systemctl restart systemd-resolved.service

mxCoder (ricardoe) wrote :

From https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636375

Another workaround:

sudo systemctl disable systemd-resolved.service

Doesn't seem to affect anything negatively.

Martin Pitt (pitti) on 2016-12-07
tags: added: resolved
Ognjen (sekil) wrote :

Hello, can you elaborate?
I don't see it resolved, rather same behaviour.

Vincent Gerris (vgerris) wrote :

@Martin Pitt : why did you mark this as resolved ??

This is a huge issue and restarting systemd-resolved did NOT fix my problem.
Also, we should not have workarounds, we should have an Ubuntu desktop that:
 - works out of the box at installation: both 16.04 AND 16.10
 - works BEFORE and AFTER an UPGRADE of 16.04
 - works without any manual work arounds or hacks

So PLEASE, come up with an actual solution that fixes this shit.
I honestly find it difficult to express how pissed off I am about the bugs because they affect both home and enterprise users.

Here is an overview with a discussion:
http://askubuntu.com/questions/838948/16-10-fail-to-resolve-dns/861991#861991

I propose to either:
 - disable all services that break normal, expected behviour: so out with systemd-resolvd if that fixes it
 - fix issues with dnsmasq and/or disable that too by default in NetworkManager

Let's try to fix this in a proper way, and have non-technical users not deal with these issues.

Anders Kaseorg (andersk) wrote :

Ognjen, Vincent: This bug is marked with the “resolved” _tag_, as in “systemd-resolved”, not as in “fixed”. The bug _status_ is still open (Confirmed).

Ognjen (sekil) wrote :

IMO, 16.04 and dnsmasq worked beautifuly for me.
I don't understand why everything that has worked, needs to be systemd-ized today.

I agree with Vincent, this is a major problem.

I am problems with the Fortclient SSL VPN client I need for work. systemd-resolved overwrites the DNS entries set by the VPN client. The connection is set up with Network-Manager. This worked fine in 16.04 and works on Arch with the same VPN client. I do not know if this is exactly the problem discusses here, but I am not able to use my VPN client software on Ubuntu 16.10.

Johan Tol (jto-joto) wrote :

The same for me today. After upgrading to 16.10 I had no connection via VPN. Deleting the line "dns=dnsmasq" from /etc/NetworkManager/Networkmanager.conf followed by 'sudo service NetworkManager restart" did the trick for me. The connection was very slow though, but I don't know if that is related to the problem. I also don't know what the "penalty" is for removing the dns=dnsmasq line.

Vincent Fortier (th0ma7) wrote :

Same here. Issue still on going with zesty 17.04 and quite painfull to deal with.

One other approach is to re-symlink resolv.conf to /run/systemd/resolve/resolv.conf but openconnect then update /run/resolvconf/resolv.conf making this a total mess.

I don't get it, this should be fairly trivial...

Markus J Schmidt (smiddy84) wrote :

Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to 16.04.0.

At the moment I am not able to work remote. Please fix this!

tags: added: xenial

OMG..this is still unresolved?
I moved to Linux mint because of this.

On Sat, Mar 11, 2017 at 3:49 PM, Markus J Schmidt <email address hidden>
wrote:

> Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to
> 16.04.0.
>
> At the moment I am not able to work remote. Please fix this!
>
> ** Tags added: xenial
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

Toomas (toomaskaevand) wrote :

I can confirm, that the issue with systemd-resolved is still very much present on Ubuntu 16.10. If Forticlient runs (split horizon), then the name resolution is very slow or nonexistent (timeouts).

Markus J Schmidt (smiddy84) wrote :
Download full text (5.5 KiB)

Here is my output for @piti:

Positive Trust Anchors:
. IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa corp home internal intranet lan local private test
Using system hostname 'Thinkpad-T440s'.
New scope on link *, protocol dns, family *
Found new link 3/wlan0
Found new link 2/eth0
Found new link 1/lo
New scope on link eth0, protocol llmnr, family AF_INET6
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Delaying llmnr transaction for 4242us.
New scope on link eth0, protocol llmnr, family AF_INET
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Delaying llmnr transaction for 51982us.
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 error=n/a
Switching to system DNS server 127.0.1.1.
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 error=n/a
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 error=n/a
Timeout reached on transaction 8616.
Retrying transaction 8616.
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Sending query packet with id 8616 on interface 2/AF_INET6.
Timeout reached on transaction 28149.
Retrying transaction 28149.
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Sending query packet with id 28149 on interface 2/AF_INET.
Got LLMNR query packet for id 8616
Sending response packet with id 8616 on interface 2/AF_INET6.
Got LLMNR query packet for id 28149
Sending response packet with id 28149 on interface 2/AF_INET.
Got LLMNR reply packet for id 8616
Processing incoming packet on transacti...

Read more...

Markus J Schmidt (smiddy84) wrote :

I can confirm that using the workaround in #6 makes the VPN operable again.

Markus J Schmidt (smiddy84) wrote :

Sorry, I mean in comment #20!

Valentin (valent1g) wrote :

Same problem here, using vpnc, Ubuntu 16.04.2.
Workaround #20 did the job.

Joe Liau (joe) wrote :

Confirmed that is still an issue on 17.04.

Comment #20 doesn't work, as that line does not exist.

Thomas (tombl) wrote :

Is this the "Additional DNS servers" and "Additional search domains" IPv4 settings? Basically, while my VPN is enabled I want to use specific a DNS server behind the VPN, but only to resolve (sub)domains underneath that search domain. This used to work for me in 16.10 and is now broken in 17.04.

ZuLu (nenominal) wrote :

Do someone plan to fix this issue?

Stephan (cpjack) wrote :

Same problem here with Ubuntu 17.04 on 2 computers.
Strange thing is, my private VPN pptp connection with my campus does not use their dns...the paid VPN (Cyberghost) works fine.

Stephan, it is response time dependent - if your campus has a slow DNS answer then the others e.g. on your normal uplink will be preferred.

I dup'ed another bug onto this, as I think they are essentially the same.

While Martin worked on the domain-restricted cases the reporter and others outlined that this is not what this bug is about.

TL;DR:
- anything (like vpn) provides new DNS servers on an extra link
  - lets call them "internal" and "external" sources of DNS info
- systemd-resolved asks ALL local interfaces concurrently and takes the fastest answer
- now we have two breaking use cases:
  - if "internal" and "external" are intentionally configured to report differently (override a DNS name to internal host) you can no more rely on getting the internal resolution
  - by asking "external" even if "internal" holds an answer is a dns leak as in [1]

At least for the VPN cases - and most here seem to be that - what we would need is a way to "disable" the DNS resolving on the original interface.

From man systemd-resolved:
Multi-label names are routed to all local interfaces that have a DNS sever configured, plus the
globally configured DNS server if there is one. [...]

If lookups are routed to multiple interfaces, the first successful response is returned (thus
effectively merging the lookup zones on all matching interfaces). If the lookup failed on all
interfaces, the last failing response is returned.

From the bug I dupped we have the output of
$ systemd-resolve --status
(There is also tcpdump data there)
Both confirms that is lists all links and their DNS
Link 4 (tun0)
      Current Scopes: DNS
[...]
         DNS Servers: 209.222.18.222
                      209.222.18.218
Link 3 (wlp3s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[...]
         DNS Servers: 192.168.0.1

Combined with the concurrent-quere-prefer-first behavior above that might be the root cause for all or most of the people chiming in here.

[1]: https://www.dnsleaktest.com/

Changed in systemd (Ubuntu):
importance: Medium → High
tags: added: zesty
Vincent Gerris (vgerris) wrote :

indeed, this is COMPLETELY broken on 17.04. It seems systemd-resolved is the only thing being used. My DNS resolving over anyconnect (openconnect) does not work AT ALL anymore. I don't know which brilliant mind decided to change things like that, knowing there are so many bugs open.

any work arounds known are greatly appreciated (disabling systemd-resolved did not help)

Tim Shannon (shannon-timothy) wrote :

Yeah currently none of the workaround mentioned in the previous comments seem to work at all for me on 17.04.

Not sure what to do at this point.

Winckler (winckler) wrote :

It's a really ugly workaround, but I'm using iptables to block connections to my ISP's DNS. I manually create and remove iptables rules using a script but at least this allows me to work remotely. I hope this get fix soon.

Vincent Gerris (vgerris) wrote :

You can still add the vpn nameserver to /etc/resolv.conf . Epic blunder by
both systemd-resolv maintainer and Ubuntu packagers for stacking a broken
configuration together for at the 3rd release. Does anyone know how to
escalate this?

On May 4, 2017 19:04, "Winckler" <email address hidden> wrote:

> It's a really ugly workaround, but I'm using iptables to block
> connections to my ISP's DNS. I manually create and remove iptables rules
> using a script but at least this allows me to work remotely. I hope this
> get fix soon.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

Tim Shannon (shannon-timothy) wrote :

Yeah, setting the nameserver in /etc/resolve.conf doesn't seem to work for me either.

tags: added: rls-aa-incoming
Changed in network-manager (Ubuntu):
status: New → Confirmed
tags: added: patch
30 comments hidden view all 110 comments
Nicholas Stommel (nstommel) wrote :
Kai-Heng Feng (kaihengfeng) wrote :

Nicholas, does the patch come from upstream? We should backport the patch into Ubuntu's NM properly, so everyone can benefit.

Nicholas Stommel (nstommel) wrote :

No, it's not an upstream patch. My patch can be applied directly to the current source on 17.04 obtained using 'apt-get source network-manager', so that would be network-manager 1.4.4-1ubuntu3 from http://us.archive.ubuntu.com/ubuntu zesty/main amd64 Packages

Kai-Heng Feng (kaihengfeng) wrote :

If that's the case, would you mind to upstream the patch?

Nicholas Stommel (nstommel) wrote :

I have upstreamed the patch at https://bugzilla.gnome.org/show_bug.cgi?id=783569 !
Hopefully this can be incorporated into future releases of network-manager :)

Nicholas Stommel (nstommel) wrote :

My apologies, it seems like this issue could have already been addressed upstream. See https://bugzilla.gnome.org/show_bug.cgi?id=783569
Anyway, I'll see if I can backport the fix provided there and whether or not it works. Sorry guys :/

Changed in systemd (Ubuntu):
assignee: nobody → Dimitri John Ledkov (xnox)
tags: removed: rls-aa-incoming
Nicholas Stommel (nstommel) wrote :

Actually I take that back. The issue is not fixed by the commit referenced on https://bugzilla.gnome.org/show_bug.cgi?id=783569 as it is already present in the current version of the network-manager. So we still have a major problem folks.

Nicholas Stommel (nstommel) wrote :

Unfortunately my patch is not a good solution for upstream application. I agree with what Beniamino Galvani mentioned, that "it is wrong to assume the connection is a VPN based on the link type, since you can have non-VPN tun/tap/gre/gretap connections as well, and they are affected by this patch." However, it seems that this issue with DNS leaks over NM-VPN connections and broken VPN split-horizon DNS using systemd-resolved still exists upstream and doesn't have a good fix.

I think this issue needs some attention and work from the Gnome-NM/systemd/Canonical devs, I've reached my limit here. :(

Housni Alaoui (reda-alaoui) wrote :

I was encountering DNS issues on Ubuntu 17.04 using OpenVPN.
Your patched NetworkManager worked for me Nicholas.
Thank you !

Nicholas Stommel (nstommel) wrote :

Hey all, so it seems like Thomas Haller at the bug thread https://bugzilla.gnome.org/show_bug.cgi?id=783569 may have actually fixed this issue upstream! Not sure how to backport the fix though, I tried and didn't have any luck, so this may be up to the package maintainers. I think this might actually be fixed though!

Nicholas Stommel (nstommel) wrote :

I have successfully backported Thomas Haller's excellent upstream solution as detailed in https://bugzilla.gnome.org/show_bug.cgi?id=783569 This took some time as things have changed quite a bit upstream, but the patch works on the current zesty 17.04 1.4.4-1ubuntu3.1 network-manager! This is a much better fix than the stopgap SetLinkDomains "." bus call based on link type I included in the previous patch. It should be reviewed for current application/submission to the package maintainers as it is basically a direct backport of Haller's fix merged upstream.

NOTE: You MUST set the ipv4.dns-priority to a negative number for the network-manager to unseat DNS configurations for other non-VPN interfaces. This patch allows for correct behavior with negative ipv4.dns-priority: "Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used." Usage of a negative dns-priority disables DNS configuration for all other interfaces, ensuring there are no DNS leaks over a VPN connection using systemd-resolved. Before Haller's bugfix, this feature did not work with systemd-resolved.

To set the ipv4.dns-priority, open the VPN connection profile you have configured through NM like so:
sudo nano /etc/NetworkManager/system-connections/<VPN-con-profile-name-here>
and adding the line (value of -42 recommended by Haller) "dns-priority=-42" so that the file contains something like:

[ipv4]
dns-priority=-42
dns-search=
method=auto

Alternatively, use the command
sudo nmcli connection modify "<VPN-con-profile-name-here>" ipv4.dns-priority -42
And you should see that the config file for that connection contains the same line as shown above. After doing so and patching/installing the patched network manager, you should not experience DNS leaks.

When I am connected to PIA's servers through network-manager-openvpn using the patched network manager and a negative ipv4.dns-priority set for my VPN connection, the output of systemd-resolved looks like this (notice that the Verizon ISP DNS server was 'unseated' and is absent for the primary wireless link wlo1):

Global
          DNSSEC NTA: 10.in-addr.arpa
                      ...(long list of NTAs omitted)...
                      test

Link 4 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

If anyone is curious about support for the routing-only domain in NM, see the following bug https://bugzilla.gnome.org/show_bug.cgi?id=746422 which is about adding support for routing-only domains for systemd-resolved (still work in progress).

Nicholas Stommel (nstommel) wrote :

After setting the ipv4.dns-priority of the VPN connection to a negative number and patching the source or installing the conveniently packaged .deb below, you should not experience DNS leaks over NM-VPN.
(Output from extended test at https://dnsleaktest.com )

Test complete

Query round Progress... Servers found
  1 ...... 1
  2 ...... 1
  3 ...... 1
  4 ...... 1
  5 ...... 1
  6 ...... 1
IP Hostname ISP Country
173.239.226.69 ip-69-226-239-173.east.us.northamericancoax.com LogicWeb Inc United States

To install the .deb package, simply use:
cd ~/Downloads && sudo dpkg -i network-manager_1.4.4-1ubuntu4_amd64.deb

NOTE: make sure apt does not replace the package with:
sudo apt-mark hold network-manager

Make sure to stop all network services and restart the network manager using:
sudo service network-manager stop
sudo service networking restart
sudo service network-manager start

To build the source and apply the patch yourself, use the following steps:

sudo apt-get build-dep network-manager
cd ~/Downloads && mkdir nm-patch && cd nm-patch
apt-get source network-manager
cd network-manager-1.4.4
cp ~/Downloads/systemd-resolved-dns-priority-fix.patch .
patch -p1 < systemd-resolved-dns-priority-fix.patch
rm systemd-resolved-dns-priority-fix.patch
dpkg-buildpackage -us -uc -b

(wait a while, it will take some time to compile)
Then install the generated network-manager_1.4.4-1ubuntu .deb package using:
cd ../ && sudo dpkg -i <deb-name>

Nicholas Stommel (nstommel) wrote :

Please test with the new patch or patched .deb and follow the steps to set negative ipv4 dns-priority. I (and lead NM-dev Thomas Haller himself) believe this resolves the bug. Thanks, and I hope this helps you all! :)

bagl0312 (bagl0312) wrote :

Hello Nicholas,
just tested the solution proposed in post #82.

My configuration is ubuntu-gnome 17.04

lsb_release -a:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

uname -a:
Linux xxxx 4.10.0-24-generic #28-Ubuntu SMP Wed Jun 14 08:14:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

The solution seems to work.

I have a working DNS tested with several openvpn configuration files and server (tested nordvpn + other personal VPNs).
No DNS leakage is observed anymore

  Thanks for your work!

Stephan (cpjack) wrote :

@Nicholas Stommel

THANKS THANKS THANKS

Hell it works !!!

Oh dear Penguin god, I was almost close to install fedora or sth else.

I owe you a beer !

Nicholas Stommel (nstommel) wrote :

@Stephan the Penguin god has not forsaken us, my friend :D
So glad it works for you guys, thanks for the nice feedback! This issue bugged me so much I sorta made it my mission haha. It's fantastic I finally got this thing sorted out with some help from the Gnome NM devs :)

I can also confirm that the latest patch fixes the problem. Thank you very much for your work!

Sampo Savola (samposavola) wrote :

Will this fix be released for 17.04 ?

no longer affects: systemd (Ubuntu)
no longer affects: systemd (Ubuntu Artful)
affects: systemd → network-manager
Changed in network-manager:
importance: Undecided → Unknown
status: New → Unknown
Dimitri John Ledkov (xnox) wrote :

@ Nicholas Stommel (nstommel)

Could you please help to update the bug description SRU template to fix this issue in 17.04?
I do not fully understand the issue at hand, but I do have access to VPN and can set VPN setting in Netowrk Manager to route all traffic through VPN. After doing that, I should check dns-leak website?! to make sure all responses come from the VPN's DNS server rather than my ISP/public DNS servers? A write up of easy steps would be nice like:
1) check dns leak website, record dns servers
2) connect to vpn
3) check dns leak website again

expected: servers in #3 should be behind vpn, and different from public dns servers listed in #1. Or some such.

Would you be able to distill testcase steps into easy steps that anybody with a VPN connection setup via network manager can reproduce? This way we will be able to validate this issue and release a stable release update.

description: updated
Changed in network-manager (Ubuntu Zesty):
status: New → Confirmed
Dimitri John Ledkov (xnox) wrote :

Also note artful has 1.8.0, thus this fix may be included there already, or e.g. will only need a simple git cherry-pick of the upstream 1.8 branch fix.

Judson W (judderwocky) wrote :

I am so sick of bugs like this in Ubuntu. Every single time I upgrade I regret it. Is this going to be available anytime this century or do I need to learn to juggle configuration scripts?

Utku Helvacı (tuxutku) wrote :

post #82 saved my day, no more dns leaks
Note: getting here took my days

Nico R (u-nico-c) wrote :

Can confirm: #82 does the trick. Thanks Nicholas, you're awesome!

Let's hope this goes into 17.04 release or at least in zesty-updates.

David Reagan (jerrac) wrote :

#82 Helped me as well. And I'm 17.04...

It'd be nice to see this fixed...

Thomas (tombl) wrote :

Does anyone know if this happens to be fixed in 17.10? I have little hope that the fix is ever going to make into 17.04...

Jordi Miralles (jmiralles) wrote :

Hi! There is a fix submitted as a patch i. The thread I have been using for a while. Works flawlessly for me.
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

13. Sep 2017 14:55 by <email address hidden>:

> Does anyone know if this happens to be fixed in 17.10? I have little
> hope that the fix is ever going to make into 17.04...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in NetworkManager:
> Unknown
> Status in network-manager package in Ubuntu:
> Confirmed
> Status in network-manager source package in Zesty:
> Confirmed
> Status in network-manager source package in Artful:
> Confirmed
>
> Bug description:
> [Impact]
>
> * NetworkManager incorrectly handles dns-priority of the VPN-like
> connections, which leads to leaking DNS queries outside of the VPN
> into the general internet.
>
> * Upstream has resolved this issue in master and 1.8 to correctly
> configure any dns backends with negative dns-priority settings.
>
> [Test Case]
>
> #FIXME#
>
> * detailed instructions how to reproduce the bug
>
> * these should allow someone who is not familiar with the affected
> package to reproduce the bug and verify that the updated package fixes
> the problem.
>
> #FIXME#
>
> [Regression Potential]
>
> * If this issue is changed DNS resolution will change, for certain
> queries, to go via VPN rather than general internet. And therefore,
> one may get new/different results or even loose access to
> resolve/access certain parts of the interent depending on what the DNS
> server on VPN chooses to respond to.
>
> [Other Info]
>
> * Original bug report
>
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed with the current Fedora release at the time reaching EOL:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/network-manager/+bug/1624317/+subscriptions

I can't even get vpn and/or socks5 to work. This is dangerous, perhaps it should be stressed that people shouldn't use 11.10 as a daily OS until this is fixed. I know, I know, me should know better. I'm just glad I checked before assuming I was in a secure tunnel. Otherwise 11.10 is working fine without any show stoppers for moi.

Thomas (tombl) wrote :

I guess I'll have to go back to 16.04 or 16.10, despite someone providing a bugfix and several people confirming it, nobody from Ubuntu seems to care. Crazy, considering how much corporate employees depend on such BASIC features like this working. Very disappointing.

Boris Malkov (hikari968) wrote :

It already looks like some kind of a tradition for ubuntu to break something critical in every single release and keep those bugs as long as possible.

demizer (jeezusjr) wrote :

Trying the above fix does not work for 17.10. This is highly unfortunate.

Ricardo (nu118yt3) wrote :

This bug still exists in Ubuntu 17.10 (ProtonVPN). In distros as Arch, Manjaro and Fedora it never happened. Is this going to 18.04 LTS as well?! Why no one cares?

Sebastien Bacher (seb128) wrote :

The corresponding GNOME bug has been marked fixed in
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=02d56ec87

that commit is in the 17.10 version, if that doesn't work then it's another issue or the upstream report should be reopened

Changed in network-manager (Ubuntu):
importance: Undecided → High
Changed in network-manager (Ubuntu Artful):
importance: Undecided → High
Changed in network-manager (Ubuntu Zesty):
importance: Undecided → High
Nicholas Stommel (nstommel) wrote :

I'm not sure about split-horizon DNS, frankly I think that is a different bug entirely. However, I have had no problems with DNS leaks over my VPN connections whatsoever on Ubuntu 17.10. The bugfix I personally requested from the NM-devs and backported to Ubuntu 17.04 (running NetworkManager v1.4.x) was effectively just patching the negative dns-priority bug related to systemd-resolved. From 'man nm-settings': "Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used." This means that DNS servers configured for the non-VPN connection will be 'unseated' and ONLY the VPN-configured DNS servers are used.

Ubuntu 17.10 is running NetworkManager v1.8.4, so Thomas Haller's merged bugfix is present and working. You MUST use the command:

'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'

or similar to actually set negative DNS priority for the VPN connection. Restart the network manager with 'sudo service network-manager restart', then connect to the VPN. Examine the output of 'systemd-resolved --status' and use the 'Extended' test on dnsleaktest.com to verify that you are not leaking DNS queries. I use openvpn, but setting negative dns priority should work for preventing DNS leaks over regular VPN connections of all kinds as a kind of 'catch-all'.

Auto-connecting to openvpn through the GUI is a little troublesome in 17.10, but this 'fix' worked for me:
https://askubuntu.com/questions/967408/how-to-automatically-connect-to-vpn-in-ubuntu-17-10/967415#967415

Ricardo (nu118yt3) wrote :

#103 did fix it

Adding:

[ipv4]
dns-priority=-42

to system-connections config file
or runing 'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'
and restarting networkmanager service
did fix dns leaking using ProtonVPN on openvpn for me, thanks.

But i didn't quite understand the problem! Is it the provided openvpn config file misconfigured or a networkmanager bug? As i said, in others distros doesn't happend!

Thomas (tombl) wrote :

The issue I had either was this bug or something else, but somehow it's apparently working in 17.10. Basically, the domain(s) of the corporate vpn that I connect to resolve over the VPN again, while everything else resolves as usual. This worked fine in 16.10, was entirely broken in 17.04 and is back to working in 17.10.

Jordi Miralles (jmiralles) wrote :

Confirming is working again in 17.10

Per former comments setting 17.10 to fix released.

Changed in network-manager (Ubuntu Artful):
status: Confirmed → Fix Released
Changed in network-manager (Ubuntu):
status: Confirmed → Fix Released
bagl0312 (bagl0312) wrote :

Hi,
I confirm that with the command:

sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42

there is not anymore DNS leakage.

However I am wondering why this command is needed, why the fix released cannot include it by default ?

Nicholas Stommel (nstommel) wrote :

@bagl0312 I agree, there really should be some kind of GUI default way to set negative DNS priority when setting up certain VPN connections. The average user shouldn't experience a nasty surprise when DNS leaks happen by default.

flux242 (flux242) wrote :

I'm not sure if setting negative priority really solves the dns leaks problem because I'm on 17.10 and I do have dns leaks. If I'm connected to my ISP over a LTE network and the connection is unstable then it could happen that DNS queries will be sent over my ISP network and not over my VPN connection. The only solution that works for me currently is
sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf:
dns=default

Delete the symlink /etc/resolv.conf
rm /etc/resolv.conf

Restart network-manager
sudo service network-manager restart

Caution! Be aware that disabling systemd-resolvd might break name resolution in VPN for some users - according to the original thread https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

Displaying first 40 and last 40 comments. View all 110 comments or add a comment.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.