systemd-resolved breaks VPN with split-horizon DNS

[Anders Kaseorg]

[Impact]

 * NetworkManager incorrectly handles dns-priority of the VPN-like connections, which leads to leaking DNS queries outside of the VPN into the general internet.

 * Upstream has resolved this issue in master and 1.8 to correctly configure any dns backends with negative dns-priority settings.

[Regression Potential]

 * If this issue is changed DNS resolution will change, for certain queries, to go via VPN rather than general internet. And therefore, one may get new/different results or even loose access to resolve/access certain parts of the interent depending on what the DNS server on VPN chooses to respond to.

[Other Info]

 * Original bug report

I use a VPN configured with network-manager-openconnect-gnome in which a split-horizon DNS setup assigns different addresses to some names inside the remote network than the addresses seen for those names from outside the remote network. However, systemd-resolved often decides to ignore the VPN’s DNS servers and use the local network’s DNS servers to resolve names (whether in the remote domain or not), breaking the split-horizon DNS.

This related bug, reported by Lennart Poettering himself, was closed with the current Fedora release at the time reaching EOL:
https://bugzilla.redhat.com/show_bug.cgi?id=1151544

[Martin Pitt (pitti)]

This matches https://github.com/systemd/systemd/issues/3421 very closely (I've worked on this a bit, but it got stalled, sorry). I think that is the exact symptom you are seeing, can you confirm?

[Anders Kaseorg (andersk)]

3421 seems like the opposite of this problem. There, queries that belong on the local network instead go to the domain-limited VPN servers. Here, queries that should go to the VPN servers (which in my case are not domain-limited) instead go to the local servers. Also, here I am using NetworkManager, not systemd-networkd.

[Anders Kaseorg (andersk)]

(I believe the requested information has been provided.)

[Martin Pitt (pitti)]

The upstream issue actually applies to both. IMHO, if you restrict a DNS server to a particular list of domains it should be used *exactly* for the given domains (only). Querying it for other domains is a privacy leak, and querying other name servers for those domains is most probably going to fail anyway and thus a waste.

> Also, here I am using NetworkManager, not systemd-networkd.

That's unrelated, as that is about resolving DNS names, not bringing up the network.

So, this does match the upstream issue, setting to triaged.

[Anders Kaseorg (andersk)]

I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

[Martin Pitt (pitti)]

> I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

This is something different then; dropping the upstream bug link. So I'm afraid this needs a more precise description of how to set this up/reproduce this, and a debug output of resolved would be helpful:

  sudo systemctl stop sytemd-resolved
  sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved

then do the operation that fails, and copy&paste the output here. Thanks!

[Martin Pitt (pitti)]

Note that in current Yakkety, NetworkManager uses dns=dnsmasq again, because until two weeks ago there was no resolved plugin for NetworkManager. That exists upstream now, but not yet in our packages, thus resolved only learned about these via dhcp-client writing the nameservers into /etc/resolv.conf and thus any association of name servers to their corresponding links was lost.

This will be fixed by the resolved plugin when NM can tell resolved about per-link DNS servers.

I think this might actually be what you mean with "split-horizon DNS". If it is, then this is fixed in yakkety, and will be fixed in z-series by using the resolved DNS plugin. If not, I don't know what you mean I'm afraid.

[Ognjen (sekil)]

Hello,

I'm having same issues with vpnc plugin and DNS servers I get from my VPN. Seems like dnsmasq doesn't try to send DNS resolve packet at all via tunnel. Once I connect to tunnel, DNS resolving is totally broken.

[Ognjen (sekil)]

This started on 16.10 btw.

[stan383 (stan383)]

The same problem here. Upgraded from 16.04 to 16.10. When using network manager for connecting to cisco VPN (network-manager-openconnect), DNS does not work at all. Even public domains are not resolved.
The only possibility for using VPN for me now is to manually add VPN's internal nameserver to /run/systemd/resolve/resolv.conf

[Matthew General (mattgen88)]

I am also affected. Broken both on a fresh install and an upgrade, dns for things behind the vpn do not resolve.

[Dennis (deki24)]

Same issue here after upgrade to 16.10.

Here is the log after connecting to VPN:
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.49.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.49 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.

Later on:
Lost too many UDP packets, downgrading feature level...
Using degraded feature set (UDP+EDNS0+DO) for DNS server 10.1.96.48.
Using feature level UDP+EDNS0+DO for transaction 35344.
Sending query packet with id 35344.
Timeout reached on transaction 29603.

[mxCoder (ricardoe)]

Hi,

I think I'm having the same issue:

Linux ricardo-N24-25BU 4.8.0-28-generic #30-Ubuntu SMP Fri Nov 11 14:03:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

Just upgraded to 16.10 last night, after bootup, I connect to a work VPN (OpenVPN + Pritunl) which was working ok so far.

route -n shows the new routes, and resolve.conf is correctly updated with the VPN DNS nameserver

Any direct query: dig, nslookup, ip route get {host} resolves correctly using the VPN DNS

But any other command: ping, telnet, mysql, etc resolves without the VPN DNS (unless instructed otherwise explicitely)

Quick solution was:

sudo systemctl restart systemd-resolved.service

[mxCoder (ricardoe)]

From https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636375

Another workaround:

sudo systemctl disable systemd-resolved.service

Doesn't seem to affect anything negatively.

[Ognjen (sekil)]

Hello, can you elaborate?
I don't see it resolved, rather same behaviour.

[Vincent Gerris (vgerris)]

@Martin Pitt : why did you mark this as resolved ??

This is a huge issue and restarting systemd-resolved did NOT fix my problem.
Also, we should not have workarounds, we should have an Ubuntu desktop that:
 - works out of the box at installation: both 16.04 AND 16.10
 - works BEFORE and AFTER an UPGRADE of 16.04
 - works without any manual work arounds or hacks

So PLEASE, come up with an actual solution that fixes this shit.
I honestly find it difficult to express how pissed off I am about the bugs because they affect both home and enterprise users.

Here is an overview with a discussion:
http://askubuntu.com/questions/838948/16-10-fail-to-resolve-dns/861991#861991

I propose to either:
 - disable all services that break normal, expected behviour: so out with systemd-resolvd if that fixes it
 - fix issues with dnsmasq and/or disable that too by default in NetworkManager

Let's try to fix this in a proper way, and have non-technical users not deal with these issues.

[Anders Kaseorg (andersk)]

Ognjen, Vincent: This bug is marked with the “resolved” _tag_, as in “systemd-resolved”, not as in “fixed”. The bug _status_ is still open (Confirmed).

[Ognjen (sekil)]

IMO, 16.04 and dnsmasq worked beautifuly for me.
I don't understand why everything that has worked, needs to be systemd-ized today.

I agree with Vincent, this is a major problem.

[Christian Dysthe (christian-dysthe)]

I am problems with the Fortclient SSL VPN client I need for work. systemd-resolved overwrites the DNS entries set by the VPN client. The connection is set up with Network-Manager. This worked fine in 16.04 and works on Arch with the same VPN client. I do not know if this is exactly the problem discusses here, but I am not able to use my VPN client software on Ubuntu 16.10.

[Johan Tol (jto-joto)]

The same for me today. After upgrading to 16.10 I had no connection via VPN. Deleting the line "dns=dnsmasq" from /etc/NetworkManager/Networkmanager.conf followed by 'sudo service NetworkManager restart" did the trick for me. The connection was very slow though, but I don't know if that is related to the problem. I also don't know what the "penalty" is for removing the dns=dnsmasq line.

[Vincent Fortier (th0ma7)]

Same here. Issue still on going with zesty 17.04 and quite painfull to deal with.

One other approach is to re-symlink resolv.conf to /run/systemd/resolve/resolv.conf but openconnect then update /run/resolvconf/resolv.conf making this a total mess.

I don't get it, this should be fairly trivial...

[Markus J Schmidt (smiddy84)]

Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to 16.04.0.

At the moment I am not able to work remote. Please fix this!

[Ognjen (sekil)]

OMG..this is still unresolved?
I moved to Linux mint because of this.

On Sat, Mar 11, 2017 at 3:49 PM, Markus J Schmidt <email address hidden>
wrote:

> Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to
> 16.04.0.
>
> At the moment I am not able to work remote. Please fix this!
>
> ** Tags added: xenial
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

[Toomas (toomaskaevand)]

I can confirm, that the issue with systemd-resolved is still very much present on Ubuntu 16.10. If Forticlient runs (split horizon), then the name resolution is very slow or nonexistent (timeouts).

[Markus J Schmidt (smiddy84)]

Here is my output for @piti:

Positive Trust Anchors:
. IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa corp home internal intranet lan local private test
Using system hostname 'Thinkpad-T440s'.
New scope on link *, protocol dns, family *
Found new link 3/wlan0
Found new link 2/eth0
Found new link 1/lo
New scope on link eth0, protocol llmnr, family AF_INET6
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Delaying llmnr transaction for 4242us.
New scope on link eth0, protocol llmnr, family AF_INET
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Delaying llmnr transaction for 51982us.
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 error=n/a
Switching to system DNS server 127.0.1.1.
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 error=n/a
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 error=n/a
Timeout reached on transaction 8616.
Retrying transaction 8616.
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Sending query packet with id 8616 on interface 2/AF_INET6.
Timeout reached on transaction 28149.
Retrying transaction 28149.
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Sending query packet with id 28149 on interface 2/AF_INET.
Got LLMNR query packet for id 8616
Sending response packet with id 8616 on interface 2/AF_INET6.
Got LLMNR query packet for id 28149
Sending response packet with id 28149 on interface 2/AF_INET.
Got LLMNR reply packet for id 8616
Processing incoming packet on transacti...

[Markus J Schmidt (smiddy84)]

I can confirm that using the workaround in #6 makes the VPN operable again.

[Markus J Schmidt (smiddy84)]

Sorry, I mean in comment #20!

[Valentin (valent1g)]

Same problem here, using vpnc, Ubuntu 16.04.2.
Workaround #20 did the job.

[Joe Liau (joe)]

Confirmed that is still an issue on 17.04.

Comment #20 doesn't work, as that line does not exist.

[Thomas (tombl)]

Is this the "Additional DNS servers" and "Additional search domains" IPv4 settings? Basically, while my VPN is enabled I want to use specific a DNS server behind the VPN, but only to resolve (sub)domains underneath that search domain. This used to work for me in 16.10 and is now broken in 17.04.

[gpothier (gpothier)]

Maybe this is related?
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1667825

[ZuLu (nenominal)]

Do someone plan to fix this issue?

[Stephan (sirstephanikus)]

Same problem here with Ubuntu 17.04 on 2 computers.
Strange thing is, my private VPN pptp connection with my campus does not use their dns...the paid VPN (Cyberghost) works fine.

[Christian Ehrhardt  (paelzer)]

Stephan, it is response time dependent - if your campus has a slow DNS answer then the others e.g. on your normal uplink will be preferred.

[Christian Ehrhardt  (paelzer)]

I dup'ed another bug onto this, as I think they are essentially the same.

While Martin worked on the domain-restricted cases the reporter and others outlined that this is not what this bug is about.

TL;DR:
- anything (like vpn) provides new DNS servers on an extra link
  - lets call them "internal" and "external" sources of DNS info
- systemd-resolved asks ALL local interfaces concurrently and takes the fastest answer
- now we have two breaking use cases:
  - if "internal" and "external" are intentionally configured to report differently (override a DNS name to internal host) you can no more rely on getting the internal resolution
  - by asking "external" even if "internal" holds an answer is a dns leak as in [1]

At least for the VPN cases - and most here seem to be that - what we would need is a way to "disable" the DNS resolving on the original interface.

From man systemd-resolved:
Multi-label names are routed to all local interfaces that have a DNS sever configured, plus the
globally configured DNS server if there is one. [...]

If lookups are routed to multiple interfaces, the first successful response is returned (thus
effectively merging the lookup zones on all matching interfaces). If the lookup failed on all
interfaces, the last failing response is returned.

From the bug I dupped we have the output of
$ systemd-resolve --status
(There is also tcpdump data there)
Both confirms that is lists all links and their DNS
Link 4 (tun0)
      Current Scopes: DNS
[...]
         DNS Servers: 209.222.18.222
                      209.222.18.218
Link 3 (wlp3s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[...]
         DNS Servers: 192.168.0.1

Combined with the concurrent-quere-prefer-first behavior above that might be the root cause for all or most of the people chiming in here.

[1]: https://www.dnsleaktest.com/

[Vincent Gerris (vgerris)]

indeed, this is COMPLETELY broken on 17.04. It seems systemd-resolved is the only thing being used. My DNS resolving over anyconnect (openconnect) does not work AT ALL anymore. I don't know which brilliant mind decided to change things like that, knowing there are so many bugs open.

any work arounds known are greatly appreciated (disabling systemd-resolved did not help)

[Tim Shannon (shannon-timothy)]

Yeah currently none of the workaround mentioned in the previous comments seem to work at all for me on 17.04.

Not sure what to do at this point.

[Winckler (winckler)]

It's a really ugly workaround, but I'm using iptables to block connections to my ISP's DNS. I manually create and remove iptables rules using a script but at least this allows me to work remotely. I hope this get fix soon.

[Vincent Gerris (vgerris)]

You can still add the vpn nameserver to /etc/resolv.conf . Epic blunder by
both systemd-resolv maintainer and Ubuntu packagers for stacking a broken
configuration together for at the 3rd release. Does anyone know how to
escalate this?

On May 4, 2017 19:04, "Winckler" <email address hidden> wrote:

> It's a really ugly workaround, but I'm using iptables to block
> connections to my ISP's DNS. I manually create and remove iptables rules
> using a script but at least this allows me to work remotely. I hope this
> get fix soon.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

[Tim Shannon (shannon-timothy)]

Yeah, setting the nameserver in /etc/resolve.conf doesn't seem to work for me either.

[Jordi Miralles (jmiralles)]

Hi,

I have been posting quite a bit of information on bug

https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1652525

As I didn't really realize there was this one open too, sorry. Maybe something is going to be useful for you.

Cheers,

J

[Vincent (dawansv)]

Here is a solution that seems to work for me.

Note that I use a simple openvpn client configuration file that I run directly from the console. I don't use a GUI for my vpn connection, but I assume you can do the same via a gui interface.

Within my openvpn client config file, I call a script called update-systemd-resolved as a replacement to the standard update-resolv-conf used previously.

That script is available here: https://github.com/jonathanio/update-systemd-resolved

After that, after connecting, when I run systemd-resolve --status I can see that a new link is added for tun0 with the proper dns entries.

But that's only a first step, because it seems that with systemd.resolved.service (I quote) "Multi-label names are routed to all local interfaces that have a DNS sever configured (...) If lookups are routed to multiple interfaces, the first successful response is returned". So basically it's still sending requests on all interfaces, including the main eth0 or whatever it is on your system, and the fastest one wins (in my case because eth0 points to my router that acts as a dns server, it always wins). To avoid leaks, we want only requests to go to the tun0 link.

Then I read that "Routing of lookups may be influenced by configuring per-interface domain names". If you read the instructions in the update-systemd-resolved, you'll see that it allows to specify dhcp-options DOMAIN-ROUTE to force a specific domain to go through that interface. You can use a single dot to mean all domains.

So basically I added dhcp-option DOMAIN-ROUTE . to my clienf config to send everything exclusively to tun0 and that seems to do the trick.

So combined with the instructions from the github page, this is what I have on my config:

# avoid dns leak when using systemd-resolved
dhcp-option DOMAIN-ROUTE .
script-security 2
setenv PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
up /etc/openvpn/update-systemd-resolved
down /etc/openvpn/update-systemd-resolved
down-pre

[Thomas M Steenholdt (tmus)]

I'm on 17.04 too and suffering from this issue for a while.

As I understand this issue, the problem may actually very well be in Network-Manager rather than in systemd-resolved, but the problem is indeed very visible with resolved.

Here's how I experience the problem (the root of my problems are a split DNS setup, just like most other people following this ticket).

This is the state of my resolved...

With no VPN connected (wireless and wifi only):

Link 7 (vpn0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 3 (wlp4s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.8.1
          DNS Domain: int.example.com

Link 2 (enp0s31f6)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.8.1
          DNS Domain: int.example.com

With VPN connected:

Link 7 (vpn0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.180.48
                      192.168.180.49
          DNS Domain: example.lan

Link 3 (wlp4s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.8.1
          DNS Domain: int.example.com

Link 2 (enp0s31f6)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.8.1
          DNS Domain: int.example.com

Now, I have one split DNS entry, testserver.example.net. On a public DNS, it will resolve to a public IP - On the example.lan DNS servers, it will resolve to a private IP.

Doing the following a couple of times is bound to sometimes return the private IP and sometimes the public IP:

systemd-resolve --flush-caches && ping -c1 testserver.example.net

So for things to work in this particular example, I'd need the 192.168.8.1 DNS to either be disabled completely or only used for int.example.com. 192.168.180.48 and 49 as provided by the VPN would somehow need to be the default/active nameserver. Note that for my VPN connection in Network Manager, I've *not* enabled the "use this connection only for resources on its own network".

In an attempt to work around this problem, I decided to configure network-manager for dnsmasq, which worked fine back in the 16.04 days. Basically the setup worked, but Network-Manager only added the VPN DNS servers for the VPN provided search domain example.lan. Needless to say this works even worse than the resolved solution, because now I get the wrong answer for testserver.example.net every time. It does seem to indicates that perhaps there's something fishy about how network-manager passes DNS servers to resolved. I have not found a way to force network-manager to completely replace the configured DNS servers for a VPN co...

[Thomas M Steenholdt (tmus)]

So I have come up with a working solution that actually solves all MY needs in this regard. Hopefully it will be of use or inspiration to some of you guys too...

Part 1 -- Switch NetworkManager to use dnsmasq (this will NOT work with resolved!)

# apt-get install dnsmasq-base

Add dns=dnsmasq to /etc/NetworkManager/NetworkManager.conf [main] section

# systemctl disable systemd-resolved
# systemctl stop systemd-resolved
# systemctl restart network-manager

Part 2 -- Modify VPN configuration (in /etc/NetworkManager/system-connections)

DNS, Routes and reverse IP for the VPN networks can be tricked to work by modifying the [ipv4] section of the VPN configuration file:

dns-search=example.lan;example2.lan;example.net # <-- make sure dns requests for these domains and all subdomains are sent to the VPN DNS servers, allowing the split DNS to work

never-default=true # <-- make sure the VPN will not be made the default route
ignore-auto-routes=true # <-- if you want to manually select the routes
route1=192.168.1.0/24 # <-- sets up a route - with reverse dns forwarding to the vpn dns server for network 1
route2=192.168.2.0/24 # <-- sets up a route - with reverse dns forwarding to the vpn dns server for network 2

[Thomas M Steenholdt (tmus)]

To clarify... I believe NetworkManager is the culprit here - or systemd-resolved is fundamentally broken (i don't have the working knowledge to guess which it is). So my comment #44 is more about getting a working system than addressing any issue with systemd and/or NetworkManager.

[Vincent (dawansv)]

Thomas:

I am not an expert on this, but as far as I can tell from the documentation you are seeing a different dns replying at times because (I quote the systemd.resolved.service doc) "Multi-label names are routed to all local interfaces that have a DNS sever configured (...) If lookups are routed to multiple interfaces, the first successful response is returned".

So basically all the dns servers defined in all of your links are fair game. DNS requests are sent to all of them at the same time and whichever replies first win the day!

My understanding is that you have to specify dhcp-options DOMAIN-ROUTE . in your openvpn connection settings to force dns requests to all domains to go through the vpn link and ignore the dns on other links.

[Thomas M Steenholdt (tmus)]

@Vincent, re the "If lookups are routed to multiple interfaces, the first successful response is returned", this is indeed the problem with systemd-resolved as I see it, as that method will never be stable for a split DNS setup... You can never reliably predict if you'll get a good or a bad IP for the connections you're currently using.

dnsmasq allows a solution to this, because NetworkManager can tell dnsmasq to use the LAN DNS for default stuff, but use the VPN DNS for lookups in the example.lan domain and 10.in-addr.arpa, for example.

The dhcp-options you mention is for a direct call to openvpn if I'm not mistaken(?). That would work if you're content with launching every VPN connection by hand. In my case, I use a bunch of different VPN clients and as such, solving this in NetworkManager is a much more universally applicable fix.

[Nicholas Stommel (nstommel)]

Please note that this patch and fix only works for Ubuntu 17.04 which relies on
systemd-resolved as a DNS/DNSSEC stub resolver, as well as an LLMNR resolver.

You also need to be using a network-manager plugin like network-manager-openvpn-gnome.
Install and configure an openvpn connection after going 'sudo apt-get install network-manager-
openvpn-gnome', importing a config file, connecting (if possible), and observing the fact that
there are DNS leaks (queries WILL be routed to your ISP) with online tools like those at
https://dnsleaktest.com. Otherwise, just know that systemd-resolved naturally leaks DNS queries
over all configured domains on all interfaces by design unless a specific system bus call is made.
In this case, the SetLinkDomains(in i ifindex, in a(sb) domains) method, if passed
the interface index followed by an array containing the "routing-only" domain "~."
(see https://manpages.debian.org/testing/systemd/systemd.network.5.en.html) and the boolean true.
But enough with the technical details, lets move on to the fix!

First, make sure you have all necessary packages to build the network manager, install with:

sudo apt update
sudo apt-get build-dep network-manager

cd ~/Documents
mkdir nm && cd nm
apt-get source network-manager
cd network-manager-1.4.4/

Copy the patch file into the network-manager-1.4.4 directory:

cp ~/Downloads/resolved-vpn-dns-leak-fix.patch .

Apply the patch with:

patch -p1 < resolved-vpn-dns-leak-fix.patch

Remove patch from source directory before compilation:

rm resolved-vpn-dns-leak-fix.patch

Compile and build .deb package for installation (this will take a while):

dpkg-buildpackage -us -uc -b

The compiled .debs should be in the parent directory you created nm:

cd ../

First, stop all network services:

sudo service network-manager stop
sudo service networking stop

Install just the patched network manager (the other .debs are not necessary):

sudo dpkg -i network-manager_1.4.4-1ubuntu3_amd64.deb

Bring network services back up:

sudo service networking start
sudo service network-manager start

Connect to standard openvpn via network-manager-openvpn GUI (or other plugin)
Search the syslog for something like:

NetworkManager[876]: <info> [1496716774.9849] systemd-resolved[0x55b0132ec2b0]: Link type is VPN
TUN or TAP, fixing DNS leak...

and verify that the VPN link (for example tun0) includes the descriptor:
 DNS Domain: ~.

using the command:

systemd-resolve --status

When compiling network manager, several bogus links are created and will show up when
you type 'systemd-resolve --status', don't worry they will disappear once you reboot.

Then open your browser, navigate to https://dnsleaktest.com and select Extended test
You should only see your VPN provider's DNS servers. For example, with PIA you should see
something like:

Test complete

Query round Progress... Servers found
  1 ...... 1
  2 ...... 1
  3 ...... 1
  4 ...... 1
  5 ...... 1
  6 ...... 1

IP Hostname ISP Country
173.239.220.5 ip-5-220-239-173.east.us.northamericancoax.com GoLightSpeed Unite...

[Nicholas Stommel (nstommel)]

The actual patch is attached above and can be applied to the source code which you can build yourself. But for your convenience, I have attached the .deb file below:

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "patch for network-manager source" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Nicholas Stommel (nstommel)]

I can confirm this works for multiple vpn connections and after wakeup from system suspend on Ubuntu 17.04. I encourage you to install the patched .deb or follow the instructions to build it from source and see for yourself. I'm honestly so glad this fixes dns leaks for using openvpn through the network manager gui on Ubuntu that I'm switching my primary machine to 17.04. :)
Please let me know if this resolves your problems with DNS leaks using a vpn via the network manager.

[Nicholas Stommel (nstommel)]

From the Debian man pages, it seems like this is not in fact a problem of systemd itself, as it allows for domain routing exclusively for dns servers on a single interface using the routing-only domain. My patch effectively just tells the NetworkManager to make a systemd bus call for the routing-only domain when the connection is a vpn tun or tap link. In fact, this feature of systemd, the routing-only domain, is a marked improvement from the glibc API, which has no equivalent concept of dns servers limited to a system link. The SetLinkDomains method of the systemd-resolved API allows for this behavior.

From SYSTEMD.NETWORK(5):

"The "routing-only" domain "~." (the tilde indicating definition of a routing domain, the dot referring to the DNS root domain which is the implied suffix of all valid DNS names) has special effect. It causes all DNS traffic which does not match another configured domain routing entry to be routed to DNS servers specified for this interface. This setting is useful to prefer a certain set of DNS servers if a link on which they are connected is available.

This setting is read by systemd-resolved.service(8). "Search domains" correspond to the domain and search entries in resolv.conf(5). Domain name routing has no equivalent in the traditional glibc API, which has no concept of domain name servers limited to a specific link."

[Jordi Miralles (jmiralles)]

Hi! Thanks for the patch Nicholas. I will upgrade to 17.04, test it and report back tonight or tomorrow at most.

[Tim Shannon (shannon-timothy)]

Not working for me, but I assume that's because I'm using network-manager-openconnect-gnome?

[Nicholas Stommel (nstommel)]

Yeah, apologies as I'm not sure what link type that openconnect uses / how to identify an openconnect link. It would be a simple matter to add a conditional for that in the file I patched, please try that. For now my patch only addresses openvpn tap or tun links, but I'm sure it could be expanded if possible. Anyone using network-manager-openvpn please test.

[bedfojo (bedfojo)]

Nicholas, thank you very much for your work on this patch.

It works correctly for me: no DNS leak detected by either https://ipleak.net or https://dnsleaktest.com for me, when both detected leaks in the unpatched version.

Running Ubuntu-MATE 17.04.

Could we perhaps get this upstreamed into NM?

[bedfojo (bedfojo)]

I should add that I'm using network-manager-openvpn and network-manager-openvpn-gnome.

[Nicholas Stommel (nstommel)]

Tim Shannon, from the comment about network-manager-openconnect-gnome, please use this updated patch to build the network manager. I added conditions for the cisco GRE and GRETAP link types, see https://en.wikipedia.org/wiki/Generic_Routing_Encapsulation and http://www.cisco.com/c/en/us/td/docs/security/security_management/cisco_security_manager/security_manager/4-4/user/guide/CSMUserGuide_wrapper/vpgredm.html#69194
so I think this might fix the issue connecting through network-manager-openconnect-gnome

Please build the network-manager with the following patch and see if DNS leaks are fixed over cisco openconnect VPN links. Thanks!

[Nicholas Stommel (nstommel)]

Anyone using Cisco PPTP/IPsec/openconnect VPN, please test the network manager with the aforementioned patch or with the updated built .deb provided here. The updated patch should address more types of VPN links. Thanks!

[Tim Shannon (shannon-timothy)]

Maybe I'm doing something wrong, but I installed the deb, and even did a full reboot, and I'm still leaking my personal IP in the DNS leak test, and am still unable to ping servers on the inside of the VPN network when connected to the ANY connect VPN.

[Nicholas Stommel (nstommel)]

Sorry to here that, I'm frankly not sure what to do about that then :/
At the very least the original patch fixes stuff for openvpn, which is good. Perhaps someone else could figure out the cisco openconnect thing.

[Jordi Miralles (jmiralles)]

Hi Nicholas,

I upgraded to 17.04, installed your patch and I can now say that dns leaks when using network-manager-openvpn + network-manager-openvpn-gnome are gone for good now. Awesome work, thanks.

[Nicholas Stommel (nstommel)]

Jordi, Sure thing, glad I could help. :)
I wonder if somebody can figure out how to help Tim with network-manager-openconnect. I tried adding two more conditions for cisco vpn gre connections but apparently it didn't work or those aren't the kind of links used. Not sure how to address that because the innards of network-manager and how it interacts with plugins is fairly complex, but maybe someone could modify the patch to account for that.
For anyone just using network-manager-openvpn, the first patch and .deb uploaded are all that is needed.

[Nicholas Stommel (nstommel)]

Tim, I have a question for you. When you connect through network-manager-openconnect-gnome, and type
systemd-resolve --status, what is your link name called? Something like 'tun0' or 'tap1' or the like?

Because I've been looking around at the openconnect wiki at http://www.infradead.org/openconnect/building.html, and it seems like openconnect requires tun/tap drivers. So theoretically, it should have worked with the original patch containing just the conditional expression:
if (link_type == NM_LINK_TYPE_TUN || link_type == NM_LINK_TYPE_TAP)
As in, this should evaluate to true as the link should be of type tun or tap.

It seems like the updated patch containing the conditional expression
if (link_type == NM_LINK_TYPE_TUN || link_type == NM_LINK_TYPE_TAP || link_type == NM_LINK_TYPE_GRE || link_type == NM_LINK_TYPE_GRETAP)
might, however, help address users of network-manager-vpnc/network-manager-vpnc-gnome. If anyone uses network-manager-vpnc/network-manager-vpnc-gnome, let me know if the newer patch containing the cases for NM_LINK_TYPE_GRE and NM_LINK_TYPE_GRETAP fixes DNS leaks for you on a Cisco PPTP/IPsec VPN. I am unable to test this out as I don't have access to any such Cisco VPN services.

Anyway, if the network manager doesn't correctly register the openconnect interface as being link type TUN or TAP, I don't really know how to fix it as that could be a problem on network-manager-openconnect's end or the result of some way it doesn't properly notify the network manager.

[Tim Shannon (shannon-timothy)]

Thanks for taking your time to work though this.

My link name is vpn0

Link 3 (vpn0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: <dns server 1>
                      <dns server 1>
          DNS Domain: ~.

Link 2 (enp30s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1

[Nicholas Stommel (nstommel)]

Huh. No, actually my patch DID work. See the line under vpn0 that says
DNS Domain: ~.
So the correct bus call was made and all dns queries SHOULD be directed to the link-specified listed DNS servers. Your problem actually appears to be that there are no link-specified dns servers.
See the line that says
DNS Servers: <dns server 1> <dns server 1>
Please try manually specifying the correct DNS servers in the network-manager-openconnect gui settings. That should fix your problem I believe.

[Tim Shannon (shannon-timothy)]

The DNS servers have always been listed under the vpn0 link when I run systemd-resolve --status, even before your patch.

I still get no internal network name resolution, even when hard coding the DNS servers in network manager.

Maybe I've got a different issue than what others are seeing, but I have confirmed that on 16.10 the vpn works correctly, resolves internal names, and doesn't leak, but on 17.04 it no longer works.

[Nicholas Stommel (nstommel)]

Huh, weird, yeah it's quite possible it's a different issue entirely, or a problem related to network-manager-openconnect. Because the routing-only domain is clearly listed as DNS Domain ~. so systemd-resolved should only send queries to the specified dns servers for the interface vpn0. Yeah...not sure sorry.

[Nicholas Stommel (nstommel)]

In reference to John Bedford's comment:

>bedfojo (commercial-johnbedford) wrote on 2017-06-06: #57
>Nicholas, thank you very much for your work on this patch.
>It works correctly for me: no DNS leak detected by either https://ipleak.net or >https://dnsleaktest.com for me, when both detected leaks in the unpatched version.
>Running Ubuntu-MATE 17.04.
>Could we perhaps get this upstreamed into NM?
>bedfojo (commercial-johnbedford) wrote on 2017-06-06: #58
>I should add that I'm using network-manager-openvpn and network-manager-openvpn-gnome.

I think it would be great if we could get this patch upstreamed into the network-manager!
I've attached a finalized version of the patch with a more informative / verbose syslog message that also accounts for cisco gre/gretap connections not in #49. Please use this patch when building network-manager for Ubuntu 17.04. I will also attach a .deb build of network-manager for easy installation and testing for anyone interested. So far, this is known to solve dns leaks with network-manager-openvpn but could also solve dns leaks for other VPNs that use TUN, TAP, or Cisco GRE network interfaces through the network-manager.
You should now see a message in your syslog when connecting that looks like the following:
NetworkManager[32636]: <info> [1496880041.6435] systemd-resolved[0x55cc602ce430]: Link #12 type is VPN TUN or TAP, fixing DNS leak...

Make sure to stop apt from replacing the patched .deb using:
sudo apt-mark hold network-manager
To verify that you are using the 'routing-only domain', use the command
systemd-resolve --status
and look for the line "DNS Domain: ~." under the VPN link number. Alternatively, check that you are not experiencing DNS leaks using the 'extended test' on https://dnsleaktest.com/

Cheers :)

[Kai-Heng Feng (kaihengfeng)]

Nicholas, does the patch come from upstream? We should backport the patch into Ubuntu's NM properly, so everyone can benefit.

[Nicholas Stommel (nstommel)]

No, it's not an upstream patch. My patch can be applied directly to the current source on 17.04 obtained using 'apt-get source network-manager', so that would be network-manager 1.4.4-1ubuntu3 from http://us.archive.ubuntu.com/ubuntu zesty/main amd64 Packages

[Kai-Heng Feng (kaihengfeng)]

If that's the case, would you mind to upstream the patch?

[Nicholas Stommel (nstommel)]

Created attachment 353426
fixes DNS leaks over some NM-VPN connections using systemd-resolved

I have patched the Network Manager to fix DNS leaks over network-manger VPN links (like those created with network-manager-openvpn) when using systemd-resolved as the default dns-manger/resolver on Ubuntu. This addresses some critical security concerns with DNS leaks over NM-VPN links.
Please see the following high-priority bug at launchpad: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
There, I attached a patch for the current version of the network manager on Ubuntu 17.04 (1.4.4-1ubuntu3 zesty). Per request of <email address hidden> on launchpad, I have patched the latest upstream source, made sure that it compiles correctly without warnings, and attached it here. So far, this is known to solve DNS leaks with network-manager-openvpn but could also solve DNS leaks for other VPNs that use TUN, TAP, or Cisco GRE network interfaces through the network-manager. It would be great to backport this fix to the current Ubuntu distribution!

[Nicholas Stommel (nstommel)]

I have upstreamed the patch at https://bugzilla.gnome.org/show_bug.cgi?id=783569 !
Hopefully this can be incorporated into future releases of network-manager :)

[Thomas Haller (thaller-1)]

there is ipv4.dns-priority setting

Quoting `man nm-settings`:

"Intra-connection DNS priority. The relative priority to be used when determining the order of DNS servers in resolv.conf. A lower value means that servers will be on top of the file. Zero selects the default value, which is 50 for VPNs and 100 for other connections. Note that the priority is to order DNS settings for multiple active connections. It does not disambiguate multiple DNS servers within the same connection profile. For that, just specify the DNS servers in the desired order. When multiple devices have configurations with the same priority, the one with an active default route will be preferred. Note that when using dns=dnsmasq the order is meaningless since dnsmasq forwards queries to all known servers at the same time. Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used."

Why is that not the right solution?

[Bgalvani (bgalvani)]

Hi,

in addition to what Thomas said, I think it is wrong to assume the
connection is a VPN based on the link type, since you can have non-VPN
tun/tap/gre/gretap connections as well, and they are affected by this
patch.

Also, it seems to me that commit [1] should already fix the leak for
VPNs that don't get the default route... did you try if git master of
NM is still affected by this bug?

If it is, can you please attach the output of 'nmcli connection show
<con-id>' for the VPN, and the output of 'systemd-resolve --status'? I
mean, without your patch, to verify why the existing commit is not
working. Thanks!

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c4864ba63f4a13e9938a978787490005f5ba48fb

[Nicholas Stommel (nstommel)]

Oh, sorry I didn't think of that :/
I suppose that commit is a better fix. I'll see if that works instead.

[Nicholas Stommel (nstommel)]

My apologies, it seems like this issue could have already been addressed upstream. See https://bugzilla.gnome.org/show_bug.cgi?id=783569
Anyway, I'll see if I can backport the fix provided there and whether or not it works. Sorry guys :/

[Nicholas Stommel (nstommel)]

Created attachment 353487
current version of nm-systemd-resolved.c in Ubuntu network manager

Wait...that commit from [1] is already found in the zesty version of network-manager 1.4.4-1ubuntu3. See attachment. This suggests that if this is still an issue on Ubuntu 17.04, the problem is still present and unsolved.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c4864ba63f4a13e9938a978787490005f5ba48fb

[Nicholas Stommel (nstommel)]

Actually I take that back. The issue is not fixed by the commit referenced on https://bugzilla.gnome.org/show_bug.cgi?id=783569 as it is already present in the current version of the network-manager. So we still have a major problem folks.

[Nicholas Stommel (nstommel)]

So the issue persists I believe. Unfortunately I just broke totally broke my installation of Ubuntu 17.04 using 'sudo make install' on the git master of NM and was unable to connect to the internet at all, so someone else is going to have to test or figure that one out. :(
I'm almost positive, however that the issue would remain. Using openvpn through network-manager-openvpn, the line "DNS Domain: ~." under the VPN link number and info with "systemd-resolved --status" is missing or has nothing after the colon. So the routing-only domain is not used for the openvpn link like tun0, and we have dns leaks. Please contact the devs at canonical assigned to https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
This is a fairly critical bug that I think needs to be worked out with other people more knowledgeable there. I'm afraid I can't help much more, sorry.

[Nicholas Stommel (nstommel)]

Unfortunately my patch is not a good solution for upstream application. I agree with what Beniamino Galvani mentioned, that "it is wrong to assume the connection is a VPN based on the link type, since you can have non-VPN tun/tap/gre/gretap connections as well, and they are affected by this patch." However, it seems that this issue with DNS leaks over NM-VPN connections and broken VPN split-horizon DNS using systemd-resolved still exists upstream and doesn't have a good fix.

I think this issue needs some attention and work from the Gnome-NM/systemd/Canonical devs, I've reached my limit here. :(

[Bgalvani (bgalvani)]

(In reply to Nick Stommel from comment #5)

Hi Nick,

thanks for submitting the patch and investigating this issue.

> So the issue persists I believe. Unfortunately I just broke totally broke my
> installation of Ubuntu 17.04 using 'sudo make install' on the git master of
> NM and was unable to connect to the internet at all, so someone else is
> going to have to test or figure that one out. :(

I guess you can easily undo the changes with 'make uninstall' and
reinstalling the deb package.

> I'm almost positive, however that the issue would remain. Using openvpn
> through network-manager-openvpn, the line "DNS Domain: ~." under the VPN
> link number and info with "systemd-resolved --status" is missing or has
> nothing after the colon. So the routing-only domain is not used for the
> openvpn link like tun0, and we have dns leaks. Please contact the devs at
> canonical assigned to
> https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
> This is a fairly critical bug that I think needs to be worked out with other
> people more knowledgeable there. I'm afraid I can't help much more, sorry.

It's not easy to understand which the actual problem is from the
launchpad bug because of some contrasting comments.

Can you please show the connection configuration and the status of
systemd-resolve when using the original NM package?

 nmcli connection show <con-id>
 systemd-resolve --status

[Housni Alaoui (reda-alaoui)]

I was encountering DNS issues on Ubuntu 17.04 using OpenVPN.
Your patched NetworkManager worked for me Nicholas.
Thank you !

[Nicholas Stommel (nstommel)]

**********
noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: home

**********
noctua@corinth:~$ nmcli connection show tun0
connection.id: tun0
connection.uuid: a61ca484-3ca9-4e88-b6e1-574b4e17ca54
connection.stable-id: --
connection.interface-name: tun0
connection.type: tun
connection.autoconnect: no
connection.autoconnect-priority: 0
connection.timestamp: 1497284475
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.autoconnect-slaves: -1 (default)
connection.secondaries:
connection.gateway-ping-timeout: 0
connection.metered: unknown
connection.lldp: -1 (default)
ipv4.method: manual
ipv4.dns:
ipv4.dns-search:
ipv4.dns-options: (default)
ipv4.dns-priority: 100
ipv4.addresses: 10.38.1.6/32
ipv4.gateway: 10.38.1.5
ipv4.routes: { ip = 10.38.1.1/32, nh = 10.38.1.5, mt = 50 }
ipv4.route-metric: 50
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-timeout: 0
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.dhcp-fqdn: --
ipv4.never-default: ...

[Nicholas Stommel (nstommel)]

After issuing the systemd-resolved API method/bus call SetLinkDomains(5 (the if_index of the vpn con-id 'tun0' in this case), ".", TRUE), the output of systemd-resolve --status for the vpn con-id 'tun0' looks like this:

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNS Domain: ~.

and there are no DNS leaks, as the routing-only domain is used.

**********
From SYSTEMD.NETWORK(5):

"The "routing-only" domain "~." (the tilde indicating definition of a routing domain, the dot referring to the DNS root domain which is the implied suffix of all valid DNS names) has special effect. It causes all DNS traffic which does not match another configured domain routing entry to be routed to DNS servers specified for this interface. This setting is useful to prefer a certain set of DNS servers if a link on which they are connected is available.

This setting is read by systemd-resolved.service(8). "Search domains" correspond to the domain and search entries in resolv.conf(5). Domain name routing has no equivalent in the traditional glibc API, which has no concept of domain name servers limited to a specific link."

**********
So the problem here is that when using network-manager-openvpn/network-manager-openvpn-gnome through the network-manager, there is no option to specify the routing-only domain, and even manually specifying the DNS servers and entering "." or "~." under "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains" has no effect (i.e. the routing-only domain is not set).

Then we have a problem where DNS queries leak by design of systemd-resolved. But fortunately, also by design, we can choose to use the'routing-only domain', which solves the problem of DNS leaks using systemd-resolved. But the network-manager does not seem to be able to issue this call for some reason, perhaps because it views "." or "~." as invalid 'Search Domains' and doesn't parse or send that information correctly to nm-systemd-resolved.c.

So...my patch is basically a stopgap measure to force the network-manager to make the bus call for the routing-only domain on a VPN connection. However, this may not be desired for all VPN connections, so obviously it's not optimal. There needs to be a change here where we can manually specify the routing-only domain for a VPN connection to prevent DNS leaks.

I think a lot of confusion on the launchpad thread is the fact that many bugs were marked duplicates of something that is actually two separate issues. Split-horizon DNS setups with openconnect are fundamentally different from DNS leaks over a 'total internet' VPN provider through which all domain-names are resolved like 'Private Internet Access' (or any number of privacy-focused VPN service providers), which I'm using through network-manager-openvpn.

[Nicholas Stommel (nstommel)]

Rather odd behavior happens when trying to specify "." or "~." in the line "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains".

**********
Here is the network config file where "." is specified under the "Search Domains" from /etc/NetworkManager/system-connections/US-East :

[connection]
id=US-East
uuid=cf291340-3c52-4347-8ce9-e609bdecec32
type=vpn
permissions=user:noctua:;
secondaries=
timestamp=1497311475

[vpn]
auth=SHA1
ca=/home/noctua/Documents/openvpn/openvpn-legacy-tcp/ca.crt
cipher=BF-CBC
comp-lzo=yes
connection-type=password
dev=tun
dev-type=tun
password-flags=1
proto-tcp=yes
remote=us-east.privateinternetaccess.com:443
remote-cert-tls=server
reneg-seconds=0
username=<my username here>
service-type=org.freedesktop.NetworkManager.openvpn

[ipv4]
dns=209.222.18.222;209.222.18.218;
dns-search=.;
ignore-auto-dns=true
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=ignore

**********
And THIS is the output of systemd-resolved for the cond-id 'tun0':

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

As you can see, the dns-search=.; is ignored entirely and "." is not passed to SetLinkDomains, the line "DNS Domain: ~." is missing.

**********
Here is the network config file where "~." is specified under the "Search Domains" from /etc/NetworkManager/system-connections/US-East :

[connection]
id=US-East
uuid=cf291340-3c52-4347-8ce9-e609bdecec32
type=vpn
permissions=user:noctua:;
secondaries=
timestamp=1497314475

[vpn]
auth=SHA1
ca=/home/noctua/Documents/openvpn/openvpn-legacy-tcp/ca.crt
cipher=BF-CBC
comp-lzo=yes
connection-type=password
dev=tun
dev-type=tun
password-flags=1
proto-tcp=yes
remote=us-east.privateinternetaccess.com:443
remote-cert-tls=server
reneg-seconds=0
username=<my username here>
service-type=org.freedesktop.NetworkManager.openvpn

[ipv4]
dns=209.222.18.222;209.222.18.218;
dns-search=~.;
ignore-auto-dns=true
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=ignore

**********
And THIS is the output of systemd-resolved for the cond-id 'tun0':

Link 9 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNS Domain: \126

**********
Something....wrong is happening when the network-manager parses the config file and sends the domains to call SetLinkDomains in nm-systemd-resolved.c. Because... '\126' is clearly not "." or "~.". In fact, it appears to be the octal value for the ASCII character "V" which...really makes no sense. The domains "." or "~." specified and correctly listed in the config file as dns-search=.; or dns-search=~.; are not being passed to SetLinkDomains in as is, which suggests a parsing error....or something in nm-systemd-resolved.c.

I think the easiest solution would be to allow "." to be parsed as a valid domain name under the dns-search label....

[Nicholas Stommel (nstommel)]

TL;DR

I think the easiest solution would be to allow "." to be parsed as a valid domain name under the dns-search label when it is sent to nm-systemd-resolved.c . That would effectively allow us to choose to use the routing-only domain, thereby solving any problems with DNS leaks over network-manager-openvpn.

[Nicholas Stommel (nstommel)]

That is, setting "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains" to contain the string "." should correctly set the routing-only domain for that connection (the network config file for the VPN connection I showed above DOES correctly contain the line "dns-search-.;", but as shown above, nothing happens. Instead, I think it gets invalidated and thrown away during parsing as a non-domain ....somewhere (I really can't find where) and is not passed to SetLinkDomains in nm-systemd-resolved.c. I have attempted to trace the parsing of this dns-search label through the network manager but honestly I have no clue where it happens. This should be an easy fix though for someone familiar with the workings of the network manager dns-manager. Thanks for your help, I believe that allowing the valid use of "." to optionally specify the routing-only domain is the optimal solution.

[Thomas Haller (thaller-1)]

NM's suggested solution to avoid DNS leaks is setting ipv4.dns-priority to a negative value. What is the shortcoming of that and why is that not sufficient to achieve what you want?

I think the shortcoming is, that a negative ipv4.dns-priority basically disables DNS configuration for all other interfaces. Contrary to "~.", which still uses other interfaces to resolve names on other interfaces if they are FQDN and the domain is configured on the other interface.

The shortcoming of ~. is that it is a missing feature that only works with systemd-resolved. It's only better then ipv4.dns-priority in case of a rather unusual usecase (like, if you want to avoid DNS leaks, but still resolve some particular FQDN).

[Nicholas Stommel (nstommel)]

Because when using systemd-resolved as a DNS manager (from Ubuntu 16.10 onward, this is NOT optional) a negative ipv4.dns-priority DOES NOT disable DNS configuration for all other interfaces. systemd-resolved could care less what priority the network-manager assigns a connection; it sends queries to all configured DNS servers on all interfaces (thereby leaking DNS queries by design).

[credit to Vincent from the launchpad thread]
"I quote the systemd.resolved.service doc) "Multi-label names are routed to all local interfaces that have a DNS sever configured (...) If lookups are routed to multiple interfaces, the first successful response is returned".

So basically all the dns servers defined in all of your links are fair game. DNS requests are sent to all of them at the same time and whichever replies first win the day!"

So literally the only solution here to prevent DNS leaks is to allow for the setting of the routing-only domain ".", which [from SYSTEMD.NETWORK(5)] "causes all DNS traffic which does not match another configured domain routing entry to be routed to DNS servers specified for this interface."

This problem is NOT solved in the current version of NM, please acknowledge the problem presented by systemd-resolved and provide a fix, such as optionally allowing for the setting of the routing-only domain as suggested above.

[Thomas Haller (thaller-1)]

(In reply to Nick Stommel from comment #13)
> systemd-resolved could care less
> what priority the network-manager assigns a connection; it sends queries to
> all configured DNS servers on all interfaces (thereby leaking DNS queries by
> design).

if the ipv4.dns-priority is negative, NM wouldn't even configure any other DNS servers in sd-resolved.

documented:

... Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used."

[Nicholas Stommel (nstommel)]

Okay...so how do I manually set the ipv4.dns-priority to be negative? Sorry, but it's not clear how to do this, a stepwise solution would be appreciated.
If this cannot be accomplished manually and the network manager already sets the ipv4.dns-priority as negative for the vpn connection-id, it's clearly not working, as there are apparent DNS leaks (as shown above and by many other users in the launchpad thread) using network-manager-openvpn with systemd-resolved as the DNS manager.

[Thomas Haller (thaller-1)]

`nmcli connection modify "$NAME" ipv4.dns-priority -42`

the default value (if left at zero) is +50 (for VPN) and +100 for other.

Beniamino had a branch to support those special domains like "~.". I don't claim it shouldn't be done. But ipv4.dns-priority should give you something similar (not identical).

[Nicholas Stommel (nstommel)]

(same results running command without backticks and with sudo)
noctua@corinth:~$ `nmcli connection modify "tun0" ipv4.dns-priority -42`

noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: home

Extended Test results from https://dnsleaktest.com/
**********
Test complete

Query round Progress... Servers found
  1 ...... 2
  2 ...... 3
  3 ...... 1
  4 ...... 2
  5 ...... 2
  6 ...... 2
IP Hostname ISP Country
71.242.0.150 none Verizon Internet Services United States
173.239.219.2 ip-2-219-239-173.east.us.northamericancoax.com LogicWeb Inc United States
71.242.0.174 none Verizon Internet Services United States
71.242.0.230 none Verizon Internet Services United States
**********

I still get DNS leaks where queries are clearly being sent to my ISP even if this command is run with/without sudo. So...that doesn't work and I believe my original claim about systemd-resolved ignoring connection priority stands.

[Bgalvani (bgalvani)]

(In reply to Nick Stommel from comment #17)
> So...that doesn't work and I believe
> my original claim about systemd-resolved ignoring connection priority stands.

This seems really strange. The code handling priorities is independent from which DNS backend (resolv.conf,dnsmasq,sd-resolved) is configured.

Please check:

1) that you have applied the change to the right connection (you can see the list of connections with 'nmcli c'). Probably 'tun0' is the name of the in-memory connection created by NM for the tun0 device created by openvpn; instead you should apply it to VPN itself.

2) that you have re-activated the VPN connection after the change

[Thomas Haller (thaller-1)]

Seems there is a bug with dns-priority together with systemd-resolved.

Please review https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=th/fix-dns-priority-for-resolved-bgo783569

[Nicholas Stommel (nstommel)]

Nope, same results unfortunately. I have tried basically everything:

noctua@corinth:~$ nmcli c
NAME UUID TYPE DEVICE
Bn80rrF8 11d6931f-8d5b-4f47-8a1e-38f33117811b 802-11-wireless wlo1
US-East cf291340-3c52-4347-8ce9-e609bdecec32 vpn wlo1
tun0 2476d73d-a1b6-4b0f-adf3-552bd095d38a tun tun0
tun0 0058ff81-1d38-4d8b-a486-a29619bb7b99 tun --
tun0 56ca187e-67bb-438e-b20a-9ed44c223619 tun --

Here's where I try every possible way to apply 'ipv4.dns-priority -42'

noctua@corinth:~$ sudo nmcli connection modify uuid ipv4.dns-priority -42
0058ff81-1d38-4d8b-a486-a29619bb7b99 56ca187e-67bb-438e-b20a-9ed44c223619
11d6931f-8d5b-4f47-8a1e-38f33117811b cf291340-3c52-4347-8ce9-e609bdecec32
2476d73d-a1b6-4b0f-adf3-552bd095d38a
noctua@corinth:~$ sudo nmcli connection modify uuid 0058ff81-1d38-4d8b-a486-a29619bb7b99 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid 56ca187e-67bb-438e-b20a-9ed44c223619 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid 2476d73d-a1b6-4b0f-adf3-552bd095d38a ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid cf291340-3c52-4347-8ce9-e609bdecec32 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify tun0 ipv4.dns-priority -42noctua@corinth:~$ sudo nmcli connection modify US-East ipv4.dns-priority -42

And then I tried reactivating the connection each time successfully through the GUI and then through nmcli, for example:

noctua@corinth:~$ sudo nmcli connection modify US-East ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection down US-East
Connection 'US-East' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/10)
noctua@corinth:~$ sudo nmcli --ask connection up US-East
A password is required to connect to 'US-East'.
Password (vpn.secrets.password):
VPN connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/12)

I even tried:
noctua@corinth:~$ sudo nmcli connection reload tun0
noctua@corinth:~$ sudo nmcli connection reload US-East

But every single time, I still I get the same results testing for DNS leaks, that DNS queries are clearly being sent to my ISP (Verizon in this case). So...this doesn't appear to work :(

I think the only real solution here to stop DNS leaks over NM-VPN connections is to provide an option to specify the routing-only domain or make "." a valid search domain name on systems that force the use of systemd-resolved (so...basically every major Linux distro including those based off Ubuntu, Arch Linux, AND the upcoming release of Debian Stretch). This is a rather serious problem, hence the kinda hackish patch I made forcing the systemd-resolved method SetLinkDomains to accept the routing-only domain ".". Ultimately, I think there needs to be the OPTION to specify the routing-only domain to prevent DNS leaks.

All those posts on the original launchpad thread, the Arch forums, and various duplicate bug threads of the original launchpad bug https://bugs.launchpad.net/ubuntu/+source...

[Nicholas Stommel (nstommel)]

Oh sorry, I didn't see your patch. I will see if I can backport the patch, thanks Thomas for looking into it! Also, um why dns-priority -42? Can it be any (reasonably sized) negative number?

[Nicholas Stommel (nstommel)]

Hmmmmm...I can't seem to get a backport working, but I somehow managed to get the git-master patched with your work from https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=th/fix-dns-priority-for-resolved-bgo783569
up and running. And...it appears like we don't have DNS leaks! Here is the output of systemd-resolved --status:

noctua@corinth:~$ systemd-resolve --status
Global
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (tun0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

and here is the output from the extended test of dnsleaktest.com:

Test complete

Query round Progress... Servers found
  1 ...... 1
  2 ...... 1
  3 ...... 1
  4 ...... 1
  5 ...... 1
  6 ...... 1
IP Hostname ISP Country
173.239.216.29 ip-29-216-239-173.east.us.northamericancoax.com LogicWeb Inc United States

...so it looks like this *might* be fixed upstream! Would it be possible for you to backport this fix for the Ubuntu package maintainers? I'm afraid I tried but nothing different happened, so this might be dependent on a number of previous commits or stuff not found in the current zesty package. It would be amazing to have a solution for this 'downstream', thanks for your work!

[Nicholas Stommel (nstommel)]

Hey all, so it seems like Thomas Haller at the bug thread https://bugzilla.gnome.org/show_bug.cgi?id=783569 may have actually fixed this issue upstream! Not sure how to backport the fix though, I tried and didn't have any luck, so this may be up to the package maintainers. I think this might actually be fixed though!

[Nicholas Stommel (nstommel)]

Edit: I should add, that this works ONLY if ipv4.dns-priority is set to be negative in the config file in .../etc/NetworkManager/system-connections/<con-name> such that it contains something like:
[ipv4]
dns-priority=-42
dns-search=
method=auto

When I properly placed "[main] dns=systemd-resolved" in the ../etc/NetworkManager/NetworkManager.conf file (I actually didn't have systemd-resolved properly configured as the dns-manager when I manually installed the NM git master with your patches applied), this is the (correct) output of systemd-resolved --status:

noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

***********

So it works! It would be amazing if you could get this fix backported successfully for the Ubuntu folks, I haven't had much luck. Thank you so much!

[Bgalvani (bgalvani)]

Branch th/fix-dns-priority-for-resolved-bgo783569 looks good to me

[Thomas Haller (thaller-1)]

merged upstream:

master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=02d56ec87c5fe5cb87ecc89eed96a7c476db7aae

nm-1-8: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=6f61497a1beb20ff2f224d163685b78c7baa76b0

[Thomas Haller (thaller-1)]

ipv4.dns-priority should now work as intended.

Then there is bug 746422 which is about adding support for routing-only domains for systemd-resolved (still work in progress).

I think this fixes the present bug.

I think the original patch is not correct, because it hard-codes an exclusive DNS configuration for VPN (which is not always wanted, and a change in behavior).
Also, it determines whether there is a VPN based on the link-type and doesn't work if you have multiple VPNs active at the same time.

Anyway, I think this issue is fixed (or the remaining parts are a duplicate of bug 746422).

Closing as fixed. Please reopen if something is missing. Thanks.

[Nicholas Stommel (nstommel)]

I have successfully backported Thomas Haller's excellent upstream solution as detailed in https://bugzilla.gnome.org/show_bug.cgi?id=783569 This took some time as things have changed quite a bit upstream, but the patch works on the current zesty 17.04 1.4.4-1ubuntu3.1 network-manager! This is a much better fix than the stopgap SetLinkDomains "." bus call based on link type I included in the previous patch. It should be reviewed for current application/submission to the package maintainers as it is basically a direct backport of Haller's fix merged upstream.

NOTE: You MUST set the ipv4.dns-priority to a negative number for the network-manager to unseat DNS configurations for other non-VPN interfaces. This patch allows for correct behavior with negative ipv4.dns-priority: "Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used." Usage of a negative dns-priority disables DNS configuration for all other interfaces, ensuring there are no DNS leaks over a VPN connection using systemd-resolved. Before Haller's bugfix, this feature did not work with systemd-resolved.

To set the ipv4.dns-priority, open the VPN connection profile you have configured through NM like so:
sudo nano /etc/NetworkManager/system-connections/<VPN-con-profile-name-here>
and adding the line (value of -42 recommended by Haller) "dns-priority=-42" so that the file contains something like:

[ipv4]
dns-priority=-42
dns-search=
method=auto

Alternatively, use the command
sudo nmcli connection modify "<VPN-con-profile-name-here>" ipv4.dns-priority -42
And you should see that the config file for that connection contains the same line as shown above. After doing so and patching/installing the patched network manager, you should not experience DNS leaks.

When I am connected to PIA's servers through network-manager-openvpn using the patched network manager and a negative ipv4.dns-priority set for my VPN connection, the output of systemd-resolved looks like this (notice that the Verizon ISP DNS server was 'unseated' and is absent for the primary wireless link wlo1):

Global
          DNSSEC NTA: 10.in-addr.arpa
                      ...(long list of NTAs omitted)...
                      test

Link 4 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

If anyone is curious about support for the routing-only domain in NM, see the following bug https://bugzilla.gnome.org/show_bug.cgi?id=746422 which is about adding support for routing-only domains for systemd-resolved (still work in progress).

[Nicholas Stommel (nstommel)]

After setting the ipv4.dns-priority of the VPN connection to a negative number and patching the source or installing the conveniently packaged .deb below, you should not experience DNS leaks over NM-VPN.
(Output from extended test at https://dnsleaktest.com )

Test complete

Query round Progress... Servers found
  1 ...... 1
  2 ...... 1
  3 ...... 1
  4 ...... 1
  5 ...... 1
  6 ...... 1
IP Hostname ISP Country
173.239.226.69 ip-69-226-239-173.east.us.northamericancoax.com LogicWeb Inc United States

To install the .deb package, simply use:
cd ~/Downloads && sudo dpkg -i network-manager_1.4.4-1ubuntu4_amd64.deb

NOTE: make sure apt does not replace the package with:
sudo apt-mark hold network-manager

Make sure to stop all network services and restart the network manager using:
sudo service network-manager stop
sudo service networking restart
sudo service network-manager start

To build the source and apply the patch yourself, use the following steps:

sudo apt-get build-dep network-manager
cd ~/Downloads && mkdir nm-patch && cd nm-patch
apt-get source network-manager
cd network-manager-1.4.4
cp ~/Downloads/systemd-resolved-dns-priority-fix.patch .
patch -p1 < systemd-resolved-dns-priority-fix.patch
rm systemd-resolved-dns-priority-fix.patch
dpkg-buildpackage -us -uc -b

(wait a while, it will take some time to compile)
Then install the generated network-manager_1.4.4-1ubuntu .deb package using:
cd ../ && sudo dpkg -i <deb-name>

[Nicholas Stommel (nstommel)]

I successfully backported the merged branch to the current version of the network-manager package in the Ubuntu 17.04 repositories! See https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317/comments/81
Thanks so much for your work :)

[Nicholas Stommel (nstommel)]

Please test with the new patch or patched .deb and follow the steps to set negative ipv4 dns-priority. I (and lead NM-dev Thomas Haller himself) believe this resolves the bug. Thanks, and I hope this helps you all! :)

[bagl0312 (bagl0312)]

Hello Nicholas,
just tested the solution proposed in post #82.

My configuration is ubuntu-gnome 17.04

lsb_release -a:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 17.04
Release: 17.04
Codename: zesty

uname -a:
Linux xxxx 4.10.0-24-generic #28-Ubuntu SMP Wed Jun 14 08:14:34 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

The solution seems to work.

I have a working DNS tested with several openvpn configuration files and server (tested nordvpn + other personal VPNs).
No DNS leakage is observed anymore

  Thanks for your work!

[Stephan (sirstephanikus)]

@Nicholas Stommel

THANKS THANKS THANKS

Hell it works !!!

Oh dear Penguin god, I was almost close to install fedora or sth else.

I owe you a beer !

[Nicholas Stommel (nstommel)]

@Stephan the Penguin god has not forsaken us, my friend :D
So glad it works for you guys, thanks for the nice feedback! This issue bugged me so much I sorta made it my mission haha. It's fantastic I finally got this thing sorted out with some help from the Gnome NM devs :)

[bedfojo (bedfojo)]

I can also confirm that the latest patch fixes the problem. Thank you very much for your work!

[Sampo Savola (samposavola)]

Will this fix be released for 17.04 ?

[Dimitri John Ledkov (xnox)]

@ Nicholas Stommel (nstommel)

Could you please help to update the bug description SRU template to fix this issue in 17.04?
I do not fully understand the issue at hand, but I do have access to VPN and can set VPN setting in Netowrk Manager to route all traffic through VPN. After doing that, I should check dns-leak website?! to make sure all responses come from the VPN's DNS server rather than my ISP/public DNS servers? A write up of easy steps would be nice like:
1) check dns leak website, record dns servers
2) connect to vpn
3) check dns leak website again

expected: servers in #3 should be behind vpn, and different from public dns servers listed in #1. Or some such.

Would you be able to distill testcase steps into easy steps that anybody with a VPN connection setup via network manager can reproduce? This way we will be able to validate this issue and release a stable release update.

[Dimitri John Ledkov (xnox)]

Also note artful has 1.8.0, thus this fix may be included there already, or e.g. will only need a simple git cherry-pick of the upstream 1.8 branch fix.

[Judson W (judderwocky)]

I am so sick of bugs like this in Ubuntu. Every single time I upgrade I regret it. Is this going to be available anytime this century or do I need to learn to juggle configuration scripts?

[Utku Helvacı (tuxutku)]

post #82 saved my day, no more dns leaks
Note: getting here took my days

[Nico R (u-nico-c)]

Can confirm: #82 does the trick. Thanks Nicholas, you're awesome!

Let's hope this goes into 17.04 release or at least in zesty-updates.

[David Reagan (jerrac)]

#82 Helped me as well. And I'm 17.04...

It'd be nice to see this fixed...

[Thomas (tombl)]

Does anyone know if this happens to be fixed in 17.10? I have little hope that the fix is ever going to make into 17.04...

[Jordi Miralles (jmiralles)]

Hi! There is a fix submitted as a patch i. The thread I have been using for a while. Works flawlessly for me.
--
Securely sent with Tutanota. Claim your encrypted mailbox today!
https://tutanota.com

13. Sep 2017 14:55 by <email address hidden>:

> Does anyone know if this happens to be fixed in 17.10? I have little
> hope that the fix is ever going to make into 17.04...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in NetworkManager:
> Unknown
> Status in network-manager package in Ubuntu:
> Confirmed
> Status in network-manager source package in Zesty:
> Confirmed
> Status in network-manager source package in Artful:
> Confirmed
>
> Bug description:
> [Impact]
>
> * NetworkManager incorrectly handles dns-priority of the VPN-like
> connections, which leads to leaking DNS queries outside of the VPN
> into the general internet.
>
> * Upstream has resolved this issue in master and 1.8 to correctly
> configure any dns backends with negative dns-priority settings.
>
> [Test Case]
>
> #FIXME#
>
> * detailed instructions how to reproduce the bug
>
> * these should allow someone who is not familiar with the affected
> package to reproduce the bug and verify that the updated package fixes
> the problem.
>
> #FIXME#
>
> [Regression Potential]
>
> * If this issue is changed DNS resolution will change, for certain
> queries, to go via VPN rather than general internet. And therefore,
> one may get new/different results or even loose access to
> resolve/access certain parts of the interent depending on what the DNS
> server on VPN chooses to respond to.
>
> [Other Info]
>
> * Original bug report
>
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed with the current Fedora release at the time reaching EOL:
> > https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/network-manager/+bug/1624317/+subscriptions

[Stephen Allen (stephen-d-allen)]

I can't even get vpn and/or socks5 to work. This is dangerous, perhaps it should be stressed that people shouldn't use 11.10 as a daily OS until this is fixed. I know, I know, me should know better. I'm just glad I checked before assuming I was in a secure tunnel. Otherwise 11.10 is working fine without any show stoppers for moi.

[Thomas (tombl)]

I guess I'll have to go back to 16.04 or 16.10, despite someone providing a bugfix and several people confirming it, nobody from Ubuntu seems to care. Crazy, considering how much corporate employees depend on such BASIC features like this working. Very disappointing.

[Boris Malkov (hikari968)]

It already looks like some kind of a tradition for ubuntu to break something critical in every single release and keep those bugs as long as possible.

[demizer (jeezusjr)]

Trying the above fix does not work for 17.10. This is highly unfortunate.

[NoName (rooster01)]

This bug still exists in Ubuntu 17.10 (ProtonVPN). In distros as Arch, Manjaro and Fedora it never happened. Is this going to 18.04 LTS as well?! Why no one cares?

[Sebastien Bacher (seb128)]

The corresponding GNOME bug has been marked fixed in
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=02d56ec87

that commit is in the 17.10 version, if that doesn't work then it's another issue or the upstream report should be reopened

[Nicholas Stommel (nstommel)]

I'm not sure about split-horizon DNS, frankly I think that is a different bug entirely. However, I have had no problems with DNS leaks over my VPN connections whatsoever on Ubuntu 17.10. The bugfix I personally requested from the NM-devs and backported to Ubuntu 17.04 (running NetworkManager v1.4.x) was effectively just patching the negative dns-priority bug related to systemd-resolved. From 'man nm-settings': "Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used." This means that DNS servers configured for the non-VPN connection will be 'unseated' and ONLY the VPN-configured DNS servers are used.

Ubuntu 17.10 is running NetworkManager v1.8.4, so Thomas Haller's merged bugfix is present and working. You MUST use the command:

'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'

or similar to actually set negative DNS priority for the VPN connection. Restart the network manager with 'sudo service network-manager restart', then connect to the VPN. Examine the output of 'systemd-resolved --status' and use the 'Extended' test on dnsleaktest.com to verify that you are not leaking DNS queries. I use openvpn, but setting negative dns priority should work for preventing DNS leaks over regular VPN connections of all kinds as a kind of 'catch-all'.

Auto-connecting to openvpn through the GUI is a little troublesome in 17.10, but this 'fix' worked for me:
https://askubuntu.com/questions/967408/how-to-automatically-connect-to-vpn-in-ubuntu-17-10/967415#967415

[NoName (rooster01)]

#103 did fix it

Adding:

[ipv4]
dns-priority=-42

to system-connections config file
or runing 'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'
and restarting networkmanager service
did fix dns leaking using ProtonVPN on openvpn for me, thanks.

But i didn't quite understand the problem! Is it the provided openvpn config file misconfigured or a networkmanager bug? As i said, in others distros doesn't happend!

[Thomas (tombl)]

The issue I had either was this bug or something else, but somehow it's apparently working in 17.10. Basically, the domain(s) of the corporate vpn that I connect to resolve over the VPN again, while everything else resolves as usual. This worked fine in 16.10, was entirely broken in 17.04 and is back to working in 17.10.

[Jordi Miralles (jmiralles)]

Confirming is working again in 17.10

[Christian Ehrhardt  (paelzer)]

Per former comments setting 17.10 to fix released.

[bagl0312 (bagl0312)]

Hi,
I confirm that with the command:

sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42

there is not anymore DNS leakage.

However I am wondering why this command is needed, why the fix released cannot include it by default ?

[Nicholas Stommel (nstommel)]

@bagl0312 I agree, there really should be some kind of GUI default way to set negative DNS priority when setting up certain VPN connections. The average user shouldn't experience a nasty surprise when DNS leaks happen by default.

[flux242 (flux242)]

I'm not sure if setting negative priority really solves the dns leaks problem because I'm on 17.10 and I do have dns leaks. If I'm connected to my ISP over a LTE network and the connection is unstable then it could happen that DNS queries will be sent over my ISP network and not over my VPN connection. The only solution that works for me currently is
sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf:
dns=default

Delete the symlink /etc/resolv.conf
rm /etc/resolv.conf

Restart network-manager
sudo service network-manager restart

Caution! Be aware that disabling systemd-resolvd might break name resolution in VPN for some users - according to the original thread https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu