systemd-resolved breaks VPN with split-horizon DNS

[Anders Kaseorg]

[Impact]

 * NetworkManager incorrectly handles dns-priority of the VPN-like connections, which leads to leaking DNS queries outside of the VPN into the general internet.

 * Upstream has resolved this issue in master and 1.8 to correctly configure any dns backends with negative dns-priority settings.

[Regression Potential]

 * If this issue is changed DNS resolution will change, for certain queries, to go via VPN rather than general internet. And therefore, one may get new/different results or even loose access to resolve/access certain parts of the interent depending on what the DNS server on VPN chooses to respond to.

[Other Info]

 * Original bug report

I use a VPN configured with network-manager-openconnect-gnome in which a split-horizon DNS setup assigns different addresses to some names inside the remote network than the addresses seen for those names from outside the remote network. However, systemd-resolved often decides to ignore the VPN’s DNS servers and use the local network’s DNS servers to resolve names (whether in the remote domain or not), breaking the split-horizon DNS.

This related bug, reported by Lennart Poettering himself, was closed with the current Fedora release at the time reaching EOL:
https://bugzilla.redhat.com/show_bug.cgi?id=1151544

[Martin Pitt (pitti)]

This matches https://github.com/systemd/systemd/issues/3421 very closely (I've worked on this a bit, but it got stalled, sorry). I think that is the exact symptom you are seeing, can you confirm?

[Anders Kaseorg (andersk)]

3421 seems like the opposite of this problem. There, queries that belong on the local network instead go to the domain-limited VPN servers. Here, queries that should go to the VPN servers (which in my case are not domain-limited) instead go to the local servers. Also, here I am using NetworkManager, not systemd-networkd.

[Anders Kaseorg (andersk)]

(I believe the requested information has been provided.)

[Martin Pitt (pitti)]

The upstream issue actually applies to both. IMHO, if you restrict a DNS server to a particular list of domains it should be used *exactly* for the given domains (only). Querying it for other domains is a privacy leak, and querying other name servers for those domains is most probably going to fail anyway and thus a waste.

> Also, here I am using NetworkManager, not systemd-networkd.

That's unrelated, as that is about resolving DNS names, not bringing up the network.

So, this does match the upstream issue, setting to triaged.

[Anders Kaseorg (andersk)]

I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

[Martin Pitt (pitti)]

> I want to be clear that in my situation the VPN’s DNS servers are _not_ marked as domain-restricted.

This is something different then; dropping the upstream bug link. So I'm afraid this needs a more precise description of how to set this up/reproduce this, and a debug output of resolved would be helpful:

  sudo systemctl stop sytemd-resolved
  sudo SYSTEMD_LOG_LEVEL=debug /lib/systemd/systemd-resolved

then do the operation that fails, and copy&paste the output here. Thanks!

[Martin Pitt (pitti)]

Note that in current Yakkety, NetworkManager uses dns=dnsmasq again, because until two weeks ago there was no resolved plugin for NetworkManager. That exists upstream now, but not yet in our packages, thus resolved only learned about these via dhcp-client writing the nameservers into /etc/resolv.conf and thus any association of name servers to their corresponding links was lost.

This will be fixed by the resolved plugin when NM can tell resolved about per-link DNS servers.

I think this might actually be what you mean with "split-horizon DNS". If it is, then this is fixed in yakkety, and will be fixed in z-series by using the resolved DNS plugin. If not, I don't know what you mean I'm afraid.

[Ognjen (sekil)]

Hello,

I'm having same issues with vpnc plugin and DNS servers I get from my VPN. Seems like dnsmasq doesn't try to send DNS resolve packet at all via tunnel. Once I connect to tunnel, DNS resolving is totally broken.

[Ognjen (sekil)]

This started on 16.10 btw.

[stan383 (stan383)]

The same problem here. Upgraded from 16.04 to 16.10. When using network manager for connecting to cisco VPN (network-manager-openconnect), DNS does not work at all. Even public domains are not resolved.
The only possibility for using VPN for me now is to manually add VPN's internal nameserver to /run/systemd/resolve/resolv.conf

[Matthew General (mattgen88)]

I am also affected. Broken both on a fresh install and an upgrade, dns for things behind the vpn do not resolve.

[Dennis (deki24)]

Same issue here after upgrade to 16.10.

Here is the log after connecting to VPN:
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.49.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.49 for transaction 29603.
Sending query packet with id 29603.
Timeout reached on transaction 29603.
Retrying transaction 29603.
Switching to system DNS server 10.1.96.48.
Cache miss for daisy.ubuntu.com IN A
Transaction 29603 for <daisy.ubuntu.com IN A> scope dns on */*.
Using feature level UDP+EDNS0+DO+LARGE for transaction 29603.
Using DNS server 10.1.96.48 for transaction 29603.
Sending query packet with id 29603.

Later on:
Lost too many UDP packets, downgrading feature level...
Using degraded feature set (UDP+EDNS0+DO) for DNS server 10.1.96.48.
Using feature level UDP+EDNS0+DO for transaction 35344.
Sending query packet with id 35344.
Timeout reached on transaction 29603.

[mxCoder (ricardoe)]

Hi,

I think I'm having the same issue:

Linux ricardo-N24-25BU 4.8.0-28-generic #30-Ubuntu SMP Fri Nov 11 14:03:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety

Just upgraded to 16.10 last night, after bootup, I connect to a work VPN (OpenVPN + Pritunl) which was working ok so far.

route -n shows the new routes, and resolve.conf is correctly updated with the VPN DNS nameserver

Any direct query: dig, nslookup, ip route get {host} resolves correctly using the VPN DNS

But any other command: ping, telnet, mysql, etc resolves without the VPN DNS (unless instructed otherwise explicitely)

Quick solution was:

sudo systemctl restart systemd-resolved.service

[mxCoder (ricardoe)]

From https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1636375

Another workaround:

sudo systemctl disable systemd-resolved.service

Doesn't seem to affect anything negatively.

[Ognjen (sekil)]

Hello, can you elaborate?
I don't see it resolved, rather same behaviour.

[Vincent Gerris (vgerris)]

@Martin Pitt : why did you mark this as resolved ??

This is a huge issue and restarting systemd-resolved did NOT fix my problem.
Also, we should not have workarounds, we should have an Ubuntu desktop that:
 - works out of the box at installation: both 16.04 AND 16.10
 - works BEFORE and AFTER an UPGRADE of 16.04
 - works without any manual work arounds or hacks

So PLEASE, come up with an actual solution that fixes this shit.
I honestly find it difficult to express how pissed off I am about the bugs because they affect both home and enterprise users.

Here is an overview with a discussion:
http://askubuntu.com/questions/838948/16-10-fail-to-resolve-dns/861991#861991

I propose to either:
 - disable all services that break normal, expected behviour: so out with systemd-resolvd if that fixes it
 - fix issues with dnsmasq and/or disable that too by default in NetworkManager

Let's try to fix this in a proper way, and have non-technical users not deal with these issues.

[Anders Kaseorg (andersk)]

Ognjen, Vincent: This bug is marked with the “resolved” _tag_, as in “systemd-resolved”, not as in “fixed”. The bug _status_ is still open (Confirmed).

[Ognjen (sekil)]

IMO, 16.04 and dnsmasq worked beautifuly for me.
I don't understand why everything that has worked, needs to be systemd-ized today.

I agree with Vincent, this is a major problem.

[Christian Dysthe (christian-dysthe)]

I am problems with the Fortclient SSL VPN client I need for work. systemd-resolved overwrites the DNS entries set by the VPN client. The connection is set up with Network-Manager. This worked fine in 16.04 and works on Arch with the same VPN client. I do not know if this is exactly the problem discusses here, but I am not able to use my VPN client software on Ubuntu 16.10.

[Johan Tol (jto-joto)]

The same for me today. After upgrading to 16.10 I had no connection via VPN. Deleting the line "dns=dnsmasq" from /etc/NetworkManager/Networkmanager.conf followed by 'sudo service NetworkManager restart" did the trick for me. The connection was very slow though, but I don't know if that is related to the problem. I also don't know what the "penalty" is for removing the dns=dnsmasq line.

[Vincent Fortier (th0ma7)]

Same here. Issue still on going with zesty 17.04 and quite painfull to deal with.

One other approach is to re-symlink resolv.conf to /run/systemd/resolve/resolv.conf but openconnect then update /run/resolvconf/resolv.conf making this a total mess.

I don't get it, this should be fairly trivial...

[Markus J Schmidt (smiddy84)]

Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to 16.04.0.

At the moment I am not able to work remote. Please fix this!

[Ognjen (sekil)]

OMG..this is still unresolved?
I moved to Linux mint because of this.

On Sat, Mar 11, 2017 at 3:49 PM, Markus J Schmidt <email address hidden>
wrote:

> Same problem with VPNC and 16.04.2. Worked before on fresh upgrade to
> 16.04.0.
>
> At the moment I am not able to work remote. Please fix this!
>
> ** Tags added: xenial
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

[Toomas (toomaskaevand)]

I can confirm, that the issue with systemd-resolved is still very much present on Ubuntu 16.10. If Forticlient runs (split horizon), then the name resolution is very slow or nonexistent (timeouts).

[Markus J Schmidt (smiddy84)]

Here is my output for @piti:

Positive Trust Anchors:
. IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa corp home internal intranet lan local private test
Using system hostname 'Thinkpad-T440s'.
New scope on link *, protocol dns, family *
Found new link 3/wlan0
Found new link 2/eth0
Found new link 1/lo
New scope on link eth0, protocol llmnr, family AF_INET6
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Delaying llmnr transaction for 4242us.
New scope on link eth0, protocol llmnr, family AF_INET
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Delaying llmnr transaction for 51982us.
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.DBus object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 error=n/a
Got message type=method_return sender=org.freedesktop.DBus destination=:1.128 object=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 error=n/a
Switching to system DNS server 127.0.1.1.
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 error=n/a
Got message type=signal sender=org.freedesktop.DBus destination=:1.128 object=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 error=n/a
Timeout reached on transaction 8616.
Retrying transaction 8616.
Excercising transaction 8616 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET6.
Sending query packet with id 8616 on interface 2/AF_INET6.
Timeout reached on transaction 28149.
Retrying transaction 28149.
Excercising transaction 28149 for <Thinkpad-T440s. IN ANY> on scope llmnr on eth0/AF_INET.
Sending query packet with id 28149 on interface 2/AF_INET.
Got LLMNR query packet for id 8616
Sending response packet with id 8616 on interface 2/AF_INET6.
Got LLMNR query packet for id 28149
Sending response packet with id 28149 on interface 2/AF_INET.
Got LLMNR reply packet for id 8616
Processing incoming packet on transacti...

[Markus J Schmidt (smiddy84)]

I can confirm that using the workaround in #6 makes the VPN operable again.

[Markus J Schmidt (smiddy84)]

Sorry, I mean in comment #20!

[Valentin (valent1g)]

Same problem here, using vpnc, Ubuntu 16.04.2.
Workaround #20 did the job.

[Joe Liau (joe)]

Confirmed that is still an issue on 17.04.

Comment #20 doesn't work, as that line does not exist.

[Thomas (tombl)]

Is this the "Additional DNS servers" and "Additional search domains" IPv4 settings? Basically, while my VPN is enabled I want to use specific a DNS server behind the VPN, but only to resolve (sub)domains underneath that search domain. This used to work for me in 16.10 and is now broken in 17.04.

[gpothier (gpothier)]

Maybe this is related?
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1667825

[ZuLu (nenominal)]

Do someone plan to fix this issue?

[Stephan (cpjack)]

Same problem here with Ubuntu 17.04 on 2 computers.
Strange thing is, my private VPN pptp connection with my campus does not use their dns...the paid VPN (Cyberghost) works fine.

[Christian Ehrhardt  (paelzer)]

Stephan, it is response time dependent - if your campus has a slow DNS answer then the others e.g. on your normal uplink will be preferred.

[Christian Ehrhardt  (paelzer)]

I dup'ed another bug onto this, as I think they are essentially the same.

While Martin worked on the domain-restricted cases the reporter and others outlined that this is not what this bug is about.

TL;DR:
- anything (like vpn) provides new DNS servers on an extra link
  - lets call them "internal" and "external" sources of DNS info
- systemd-resolved asks ALL local interfaces concurrently and takes the fastest answer
- now we have two breaking use cases:
  - if "internal" and "external" are intentionally configured to report differently (override a DNS name to internal host) you can no more rely on getting the internal resolution
  - by asking "external" even if "internal" holds an answer is a dns leak as in [1]

At least for the VPN cases - and most here seem to be that - what we would need is a way to "disable" the DNS resolving on the original interface.

From man systemd-resolved:
Multi-label names are routed to all local interfaces that have a DNS sever configured, plus the
globally configured DNS server if there is one. [...]

If lookups are routed to multiple interfaces, the first successful response is returned (thus
effectively merging the lookup zones on all matching interfaces). If the lookup failed on all
interfaces, the last failing response is returned.

From the bug I dupped we have the output of
$ systemd-resolve --status
(There is also tcpdump data there)
Both confirms that is lists all links and their DNS
Link 4 (tun0)
      Current Scopes: DNS
[...]
         DNS Servers: 209.222.18.222
                      209.222.18.218
Link 3 (wlp3s0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
[...]
         DNS Servers: 192.168.0.1

Combined with the concurrent-quere-prefer-first behavior above that might be the root cause for all or most of the people chiming in here.

[1]: https://www.dnsleaktest.com/

[Vincent Gerris (vgerris)]

indeed, this is COMPLETELY broken on 17.04. It seems systemd-resolved is the only thing being used. My DNS resolving over anyconnect (openconnect) does not work AT ALL anymore. I don't know which brilliant mind decided to change things like that, knowing there are so many bugs open.

any work arounds known are greatly appreciated (disabling systemd-resolved did not help)

[Tim Shannon (shannon-timothy)]

Yeah currently none of the workaround mentioned in the previous comments seem to work at all for me on 17.04.

Not sure what to do at this point.

[Winckler (winckler)]

It's a really ugly workaround, but I'm using iptables to block connections to my ISP's DNS. I manually create and remove iptables rules using a script but at least this allows me to work remotely. I hope this get fix soon.

[Vincent Gerris (vgerris)]

You can still add the vpn nameserver to /etc/resolv.conf . Epic blunder by
both systemd-resolv maintainer and Ubuntu packagers for stacking a broken
configuration together for at the 3rd release. Does anyone know how to
escalate this?

On May 4, 2017 19:04, "Winckler" <email address hidden> wrote:

> It's a really ugly workaround, but I'm using iptables to block
> connections to my ISP's DNS. I manually create and remove iptables rules
> using a script but at least this allows me to work remotely. I hope this
> get fix soon.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1624317
>
> Title:
> systemd-resolved breaks VPN with split-horizon DNS
>
> Status in systemd:
> New
> Status in systemd package in Ubuntu:
> Confirmed
>
> Bug description:
> I use a VPN configured with network-manager-openconnect-gnome in which
> a split-horizon DNS setup assigns different addresses to some names
> inside the remote network than the addresses seen for those names from
> outside the remote network. However, systemd-resolved often decides
> to ignore the VPN’s DNS servers and use the local network’s DNS
> servers to resolve names (whether in the remote domain or not),
> breaking the split-horizon DNS.
>
> This related bug, reported by Lennart Poettering himself, was closed
> with the current Fedora release at the time reaching EOL:
> https://bugzilla.redhat.com/show_bug.cgi?id=1151544
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/systemd/+bug/1624317/+subscriptions
>

[Tim Shannon (shannon-timothy)]

Yeah, setting the nameserver in /etc/resolve.conf doesn't seem to work for me either.

[Nicholas Stommel (nstommel)]

Created attachment 353426
fixes DNS leaks over some NM-VPN connections using systemd-resolved

I have patched the Network Manager to fix DNS leaks over network-manger VPN links (like those created with network-manager-openvpn) when using systemd-resolved as the default dns-manger/resolver on Ubuntu. This addresses some critical security concerns with DNS leaks over NM-VPN links.
Please see the following high-priority bug at launchpad: https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
There, I attached a patch for the current version of the network manager on Ubuntu 17.04 (1.4.4-1ubuntu3 zesty). Per request of <email address hidden> on launchpad, I have patched the latest upstream source, made sure that it compiles correctly without warnings, and attached it here. So far, this is known to solve DNS leaks with network-manager-openvpn but could also solve DNS leaks for other VPNs that use TUN, TAP, or Cisco GRE network interfaces through the network-manager. It would be great to backport this fix to the current Ubuntu distribution!

[Thomas Haller (thaller-1)]

there is ipv4.dns-priority setting

Quoting `man nm-settings`:

"Intra-connection DNS priority. The relative priority to be used when determining the order of DNS servers in resolv.conf. A lower value means that servers will be on top of the file. Zero selects the default value, which is 50 for VPNs and 100 for other connections. Note that the priority is to order DNS settings for multiple active connections. It does not disambiguate multiple DNS servers within the same connection profile. For that, just specify the DNS servers in the desired order. When multiple devices have configurations with the same priority, the one with an active default route will be preferred. Note that when using dns=dnsmasq the order is meaningless since dnsmasq forwards queries to all known servers at the same time. Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used."

Why is that not the right solution?

[Bgalvani (bgalvani)]

Hi,

in addition to what Thomas said, I think it is wrong to assume the
connection is a VPN based on the link type, since you can have non-VPN
tun/tap/gre/gretap connections as well, and they are affected by this
patch.

Also, it seems to me that commit [1] should already fix the leak for
VPNs that don't get the default route... did you try if git master of
NM is still affected by this bug?

If it is, can you please attach the output of 'nmcli connection show
<con-id>' for the VPN, and the output of 'systemd-resolve --status'? I
mean, without your patch, to verify why the existing commit is not
working. Thanks!

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c4864ba63f4a13e9938a978787490005f5ba48fb

[Nicholas Stommel (nstommel)]

Oh, sorry I didn't think of that :/
I suppose that commit is a better fix. I'll see if that works instead.

[Nicholas Stommel (nstommel)]

Created attachment 353487
current version of nm-systemd-resolved.c in Ubuntu network manager

Wait...that commit from [1] is already found in the zesty version of network-manager 1.4.4-1ubuntu3. See attachment. This suggests that if this is still an issue on Ubuntu 17.04, the problem is still present and unsolved.

[1] https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=c4864ba63f4a13e9938a978787490005f5ba48fb

[Nicholas Stommel (nstommel)]

So the issue persists I believe. Unfortunately I just broke totally broke my installation of Ubuntu 17.04 using 'sudo make install' on the git master of NM and was unable to connect to the internet at all, so someone else is going to have to test or figure that one out. :(
I'm almost positive, however that the issue would remain. Using openvpn through network-manager-openvpn, the line "DNS Domain: ~." under the VPN link number and info with "systemd-resolved --status" is missing or has nothing after the colon. So the routing-only domain is not used for the openvpn link like tun0, and we have dns leaks. Please contact the devs at canonical assigned to https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
This is a fairly critical bug that I think needs to be worked out with other people more knowledgeable there. I'm afraid I can't help much more, sorry.

[Bgalvani (bgalvani)]

(In reply to Nick Stommel from comment #5)

Hi Nick,

thanks for submitting the patch and investigating this issue.

> So the issue persists I believe. Unfortunately I just broke totally broke my
> installation of Ubuntu 17.04 using 'sudo make install' on the git master of
> NM and was unable to connect to the internet at all, so someone else is
> going to have to test or figure that one out. :(

I guess you can easily undo the changes with 'make uninstall' and
reinstalling the deb package.

> I'm almost positive, however that the issue would remain. Using openvpn
> through network-manager-openvpn, the line "DNS Domain: ~." under the VPN
> link number and info with "systemd-resolved --status" is missing or has
> nothing after the colon. So the routing-only domain is not used for the
> openvpn link like tun0, and we have dns leaks. Please contact the devs at
> canonical assigned to
> https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317
> This is a fairly critical bug that I think needs to be worked out with other
> people more knowledgeable there. I'm afraid I can't help much more, sorry.

It's not easy to understand which the actual problem is from the
launchpad bug because of some contrasting comments.

Can you please show the connection configuration and the status of
systemd-resolve when using the original NM package?

 nmcli connection show <con-id>
 systemd-resolve --status

[Nicholas Stommel (nstommel)]

**********
noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: home

**********
noctua@corinth:~$ nmcli connection show tun0
connection.id: tun0
connection.uuid: a61ca484-3ca9-4e88-b6e1-574b4e17ca54
connection.stable-id: --
connection.interface-name: tun0
connection.type: tun
connection.autoconnect: no
connection.autoconnect-priority: 0
connection.timestamp: 1497284475
connection.read-only: no
connection.permissions:
connection.zone: --
connection.master: --
connection.slave-type: --
connection.autoconnect-slaves: -1 (default)
connection.secondaries:
connection.gateway-ping-timeout: 0
connection.metered: unknown
connection.lldp: -1 (default)
ipv4.method: manual
ipv4.dns:
ipv4.dns-search:
ipv4.dns-options: (default)
ipv4.dns-priority: 100
ipv4.addresses: 10.38.1.6/32
ipv4.gateway: 10.38.1.5
ipv4.routes: { ip = 10.38.1.1/32, nh = 10.38.1.5, mt = 50 }
ipv4.route-metric: 50
ipv4.ignore-auto-routes: no
ipv4.ignore-auto-dns: no
ipv4.dhcp-client-id: --
ipv4.dhcp-timeout: 0
ipv4.dhcp-send-hostname: yes
ipv4.dhcp-hostname: --
ipv4.dhcp-fqdn: --
ipv4.never-default: ...

[Nicholas Stommel (nstommel)]

After issuing the systemd-resolved API method/bus call SetLinkDomains(5 (the if_index of the vpn con-id 'tun0' in this case), ".", TRUE), the output of systemd-resolve --status for the vpn con-id 'tun0' looks like this:

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNS Domain: ~.

and there are no DNS leaks, as the routing-only domain is used.

**********
From SYSTEMD.NETWORK(5):

"The "routing-only" domain "~." (the tilde indicating definition of a routing domain, the dot referring to the DNS root domain which is the implied suffix of all valid DNS names) has special effect. It causes all DNS traffic which does not match another configured domain routing entry to be routed to DNS servers specified for this interface. This setting is useful to prefer a certain set of DNS servers if a link on which they are connected is available.

This setting is read by systemd-resolved.service(8). "Search domains" correspond to the domain and search entries in resolv.conf(5). Domain name routing has no equivalent in the traditional glibc API, which has no concept of domain name servers limited to a specific link."

**********
So the problem here is that when using network-manager-openvpn/network-manager-openvpn-gnome through the network-manager, there is no option to specify the routing-only domain, and even manually specifying the DNS servers and entering "." or "~." under "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains" has no effect (i.e. the routing-only domain is not set).

Then we have a problem where DNS queries leak by design of systemd-resolved. But fortunately, also by design, we can choose to use the'routing-only domain', which solves the problem of DNS leaks using systemd-resolved. But the network-manager does not seem to be able to issue this call for some reason, perhaps because it views "." or "~." as invalid 'Search Domains' and doesn't parse or send that information correctly to nm-systemd-resolved.c.

So...my patch is basically a stopgap measure to force the network-manager to make the bus call for the routing-only domain on a VPN connection. However, this may not be desired for all VPN connections, so obviously it's not optimal. There needs to be a change here where we can manually specify the routing-only domain for a VPN connection to prevent DNS leaks.

I think a lot of confusion on the launchpad thread is the fact that many bugs were marked duplicates of something that is actually two separate issues. Split-horizon DNS setups with openconnect are fundamentally different from DNS leaks over a 'total internet' VPN provider through which all domain-names are resolved like 'Private Internet Access' (or any number of privacy-focused VPN service providers), which I'm using through network-manager-openvpn.

[Nicholas Stommel (nstommel)]

Rather odd behavior happens when trying to specify "." or "~." in the line "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains".

**********
Here is the network config file where "." is specified under the "Search Domains" from /etc/NetworkManager/system-connections/US-East :

[connection]
id=US-East
uuid=cf291340-3c52-4347-8ce9-e609bdecec32
type=vpn
permissions=user:noctua:;
secondaries=
timestamp=1497311475

[vpn]
auth=SHA1
ca=/home/noctua/Documents/openvpn/openvpn-legacy-tcp/ca.crt
cipher=BF-CBC
comp-lzo=yes
connection-type=password
dev=tun
dev-type=tun
password-flags=1
proto-tcp=yes
remote=us-east.privateinternetaccess.com:443
remote-cert-tls=server
reneg-seconds=0
username=<my username here>
service-type=org.freedesktop.NetworkManager.openvpn

[ipv4]
dns=209.222.18.222;209.222.18.218;
dns-search=.;
ignore-auto-dns=true
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=ignore

**********
And THIS is the output of systemd-resolved for the cond-id 'tun0':

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

As you can see, the dns-search=.; is ignored entirely and "." is not passed to SetLinkDomains, the line "DNS Domain: ~." is missing.

**********
Here is the network config file where "~." is specified under the "Search Domains" from /etc/NetworkManager/system-connections/US-East :

[connection]
id=US-East
uuid=cf291340-3c52-4347-8ce9-e609bdecec32
type=vpn
permissions=user:noctua:;
secondaries=
timestamp=1497314475

[vpn]
auth=SHA1
ca=/home/noctua/Documents/openvpn/openvpn-legacy-tcp/ca.crt
cipher=BF-CBC
comp-lzo=yes
connection-type=password
dev=tun
dev-type=tun
password-flags=1
proto-tcp=yes
remote=us-east.privateinternetaccess.com:443
remote-cert-tls=server
reneg-seconds=0
username=<my username here>
service-type=org.freedesktop.NetworkManager.openvpn

[ipv4]
dns=209.222.18.222;209.222.18.218;
dns-search=~.;
ignore-auto-dns=true
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=ignore

**********
And THIS is the output of systemd-resolved for the cond-id 'tun0':

Link 9 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNS Domain: \126

**********
Something....wrong is happening when the network-manager parses the config file and sends the domains to call SetLinkDomains in nm-systemd-resolved.c. Because... '\126' is clearly not "." or "~.". In fact, it appears to be the octal value for the ASCII character "V" which...really makes no sense. The domains "." or "~." specified and correctly listed in the config file as dns-search=.; or dns-search=~.; are not being passed to SetLinkDomains in as is, which suggests a parsing error....or something in nm-systemd-resolved.c.

I think the easiest solution would be to allow "." to be parsed as a valid domain name under the dns-search label....

[Nicholas Stommel (nstommel)]

TL;DR

I think the easiest solution would be to allow "." to be parsed as a valid domain name under the dns-search label when it is sent to nm-systemd-resolved.c . That would effectively allow us to choose to use the routing-only domain, thereby solving any problems with DNS leaks over network-manager-openvpn.

[Nicholas Stommel (nstommel)]

That is, setting "Edit Connections"->"<VPN Connection Name>"->"IPv4 Settings"->"Method=Automatic (VPN) addresses only"->"Search Domains" to contain the string "." should correctly set the routing-only domain for that connection (the network config file for the VPN connection I showed above DOES correctly contain the line "dns-search-.;", but as shown above, nothing happens. Instead, I think it gets invalidated and thrown away during parsing as a non-domain ....somewhere (I really can't find where) and is not passed to SetLinkDomains in nm-systemd-resolved.c. I have attempted to trace the parsing of this dns-search label through the network manager but honestly I have no clue where it happens. This should be an easy fix though for someone familiar with the workings of the network manager dns-manager. Thanks for your help, I believe that allowing the valid use of "." to optionally specify the routing-only domain is the optimal solution.

[Thomas Haller (thaller-1)]

NM's suggested solution to avoid DNS leaks is setting ipv4.dns-priority to a negative value. What is the shortcoming of that and why is that not sufficient to achieve what you want?

I think the shortcoming is, that a negative ipv4.dns-priority basically disables DNS configuration for all other interfaces. Contrary to "~.", which still uses other interfaces to resolve names on other interfaces if they are FQDN and the domain is configured on the other interface.

The shortcoming of ~. is that it is a missing feature that only works with systemd-resolved. It's only better then ipv4.dns-priority in case of a rather unusual usecase (like, if you want to avoid DNS leaks, but still resolve some particular FQDN).

[Nicholas Stommel (nstommel)]

Because when using systemd-resolved as a DNS manager (from Ubuntu 16.10 onward, this is NOT optional) a negative ipv4.dns-priority DOES NOT disable DNS configuration for all other interfaces. systemd-resolved could care less what priority the network-manager assigns a connection; it sends queries to all configured DNS servers on all interfaces (thereby leaking DNS queries by design).

[credit to Vincent from the launchpad thread]
"I quote the systemd.resolved.service doc) "Multi-label names are routed to all local interfaces that have a DNS sever configured (...) If lookups are routed to multiple interfaces, the first successful response is returned".

So basically all the dns servers defined in all of your links are fair game. DNS requests are sent to all of them at the same time and whichever replies first win the day!"

So literally the only solution here to prevent DNS leaks is to allow for the setting of the routing-only domain ".", which [from SYSTEMD.NETWORK(5)] "causes all DNS traffic which does not match another configured domain routing entry to be routed to DNS servers specified for this interface."

This problem is NOT solved in the current version of NM, please acknowledge the problem presented by systemd-resolved and provide a fix, such as optionally allowing for the setting of the routing-only domain as suggested above.

[Thomas Haller (thaller-1)]

(In reply to Nick Stommel from comment #13)
> systemd-resolved could care less
> what priority the network-manager assigns a connection; it sends queries to
> all configured DNS servers on all interfaces (thereby leaking DNS queries by
> design).

if the ipv4.dns-priority is negative, NM wouldn't even configure any other DNS servers in sd-resolved.

documented:

... Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used."

[Nicholas Stommel (nstommel)]

Okay...so how do I manually set the ipv4.dns-priority to be negative? Sorry, but it's not clear how to do this, a stepwise solution would be appreciated.
If this cannot be accomplished manually and the network manager already sets the ipv4.dns-priority as negative for the vpn connection-id, it's clearly not working, as there are apparent DNS leaks (as shown above and by many other users in the launchpad thread) using network-manager-openvpn with systemd-resolved as the DNS manager.

[Thomas Haller (thaller-1)]

`nmcli connection modify "$NAME" ipv4.dns-priority -42`

the default value (if left at zero) is +50 (for VPN) and +100 for other.

Beniamino had a branch to support those special domains like "~.". I don't claim it shouldn't be done. But ipv4.dns-priority should give you something similar (not identical).

[Nicholas Stommel (nstommel)]

(same results running command without backticks and with sudo)
noctua@corinth:~$ `nmcli connection modify "tun0" ipv4.dns-priority -42`

noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 192.168.1.1
          DNS Domain: home

Extended Test results from https://dnsleaktest.com/
**********
Test complete

Query round Progress... Servers found
  1 ...... 2
  2 ...... 3
  3 ...... 1
  4 ...... 2
  5 ...... 2
  6 ...... 2
IP Hostname ISP Country
71.242.0.150 none Verizon Internet Services United States
173.239.219.2 ip-2-219-239-173.east.us.northamericancoax.com LogicWeb Inc United States
71.242.0.174 none Verizon Internet Services United States
71.242.0.230 none Verizon Internet Services United States
**********

I still get DNS leaks where queries are clearly being sent to my ISP even if this command is run with/without sudo. So...that doesn't work and I believe my original claim about systemd-resolved ignoring connection priority stands.

[Bgalvani (bgalvani)]

(In reply to Nick Stommel from comment #17)
> So...that doesn't work and I believe
> my original claim about systemd-resolved ignoring connection priority stands.

This seems really strange. The code handling priorities is independent from which DNS backend (resolv.conf,dnsmasq,sd-resolved) is configured.

Please check:

1) that you have applied the change to the right connection (you can see the list of connections with 'nmcli c'). Probably 'tun0' is the name of the in-memory connection created by NM for the tun0 device created by openvpn; instead you should apply it to VPN itself.

2) that you have re-activated the VPN connection after the change

[Thomas Haller (thaller-1)]

Seems there is a bug with dns-priority together with systemd-resolved.

Please review https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=th/fix-dns-priority-for-resolved-bgo783569

[Nicholas Stommel (nstommel)]

Nope, same results unfortunately. I have tried basically everything:

noctua@corinth:~$ nmcli c
NAME UUID TYPE DEVICE
Bn80rrF8 11d6931f-8d5b-4f47-8a1e-38f33117811b 802-11-wireless wlo1
US-East cf291340-3c52-4347-8ce9-e609bdecec32 vpn wlo1
tun0 2476d73d-a1b6-4b0f-adf3-552bd095d38a tun tun0
tun0 0058ff81-1d38-4d8b-a486-a29619bb7b99 tun --
tun0 56ca187e-67bb-438e-b20a-9ed44c223619 tun --

Here's where I try every possible way to apply 'ipv4.dns-priority -42'

noctua@corinth:~$ sudo nmcli connection modify uuid ipv4.dns-priority -42
0058ff81-1d38-4d8b-a486-a29619bb7b99 56ca187e-67bb-438e-b20a-9ed44c223619
11d6931f-8d5b-4f47-8a1e-38f33117811b cf291340-3c52-4347-8ce9-e609bdecec32
2476d73d-a1b6-4b0f-adf3-552bd095d38a
noctua@corinth:~$ sudo nmcli connection modify uuid 0058ff81-1d38-4d8b-a486-a29619bb7b99 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid 56ca187e-67bb-438e-b20a-9ed44c223619 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid 2476d73d-a1b6-4b0f-adf3-552bd095d38a ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify uuid cf291340-3c52-4347-8ce9-e609bdecec32 ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection modify tun0 ipv4.dns-priority -42noctua@corinth:~$ sudo nmcli connection modify US-East ipv4.dns-priority -42

And then I tried reactivating the connection each time successfully through the GUI and then through nmcli, for example:

noctua@corinth:~$ sudo nmcli connection modify US-East ipv4.dns-priority -42
noctua@corinth:~$ sudo nmcli connection down US-East
Connection 'US-East' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/10)
noctua@corinth:~$ sudo nmcli --ask connection up US-East
A password is required to connect to 'US-East'.
Password (vpn.secrets.password):
VPN connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/12)

I even tried:
noctua@corinth:~$ sudo nmcli connection reload tun0
noctua@corinth:~$ sudo nmcli connection reload US-East

But every single time, I still I get the same results testing for DNS leaks, that DNS queries are clearly being sent to my ISP (Verizon in this case). So...this doesn't appear to work :(

I think the only real solution here to stop DNS leaks over NM-VPN connections is to provide an option to specify the routing-only domain or make "." a valid search domain name on systems that force the use of systemd-resolved (so...basically every major Linux distro including those based off Ubuntu, Arch Linux, AND the upcoming release of Debian Stretch). This is a rather serious problem, hence the kinda hackish patch I made forcing the systemd-resolved method SetLinkDomains to accept the routing-only domain ".". Ultimately, I think there needs to be the OPTION to specify the routing-only domain to prevent DNS leaks.

All those posts on the original launchpad thread, the Arch forums, and various duplicate bug threads of the original launchpad bug https://bugs.launchpad.net/ubuntu/+source...

[Nicholas Stommel (nstommel)]

Oh sorry, I didn't see your patch. I will see if I can backport the patch, thanks Thomas for looking into it! Also, um why dns-priority -42? Can it be any (reasonably sized) negative number?

[Nicholas Stommel (nstommel)]

Hmmmmm...I can't seem to get a backport working, but I somehow managed to get the git-master patched with your work from https://cgit.freedesktop.org/NetworkManager/NetworkManager/log/?h=th/fix-dns-priority-for-resolved-bgo783569
up and running. And...it appears like we don't have DNS leaks! Here is the output of systemd-resolved --status:

noctua@corinth:~$ systemd-resolve --status
Global
         DNS Servers: 209.222.18.222
                      209.222.18.218
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 3 (tun0)
      Current Scopes: none
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

and here is the output from the extended test of dnsleaktest.com:

Test complete

Query round Progress... Servers found
  1 ...... 1
  2 ...... 1
  3 ...... 1
  4 ...... 1
  5 ...... 1
  6 ...... 1
IP Hostname ISP Country
173.239.216.29 ip-29-216-239-173.east.us.northamericancoax.com LogicWeb Inc United States

...so it looks like this *might* be fixed upstream! Would it be possible for you to backport this fix for the Ubuntu package maintainers? I'm afraid I tried but nothing different happened, so this might be dependent on a number of previous commits or stuff not found in the current zesty package. It would be amazing to have a solution for this 'downstream', thanks for your work!

[Nicholas Stommel (nstommel)]

Edit: I should add, that this works ONLY if ipv4.dns-priority is set to be negative in the config file in .../etc/NetworkManager/system-connections/<con-name> such that it contains something like:
[ipv4]
dns-priority=-42
dns-search=
method=auto

When I properly placed "[main] dns=systemd-resolved" in the ../etc/NetworkManager/NetworkManager.conf file (I actually didn't have systemd-resolved properly configured as the dns-manager when I manually installed the NM git master with your patches applied), this is the (correct) output of systemd-resolved --status:

noctua@corinth:~$ systemd-resolve --status
Global
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 5 (tun0)
      Current Scopes: DNS
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
         DNS Servers: 209.222.18.222
                      209.222.18.218

Link 2 (wlo1)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
       LLMNR setting: yes
MulticastDNS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

***********

So it works! It would be amazing if you could get this fix backported successfully for the Ubuntu folks, I haven't had much luck. Thank you so much!

[Bgalvani (bgalvani)]

Branch th/fix-dns-priority-for-resolved-bgo783569 looks good to me

[Thomas Haller (thaller-1)]

merged upstream:

master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=02d56ec87c5fe5cb87ecc89eed96a7c476db7aae

nm-1-8: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=6f61497a1beb20ff2f224d163685b78c7baa76b0

[Thomas Haller (thaller-1)]

ipv4.dns-priority should now work as intended.

Then there is bug 746422 which is about adding support for routing-only domains for systemd-resolved (still work in progress).

I think this fixes the present bug.

I think the original patch is not correct, because it hard-codes an exclusive DNS configuration for VPN (which is not always wanted, and a change in behavior).
Also, it determines whether there is a VPN based on the link-type and doesn't work if you have multiple VPNs active at the same time.

Anyway, I think this issue is fixed (or the remaining parts are a duplicate of bug 746422).

Closing as fixed. Please reopen if something is missing. Thanks.

[Nicholas Stommel (nstommel)]

I successfully backported the merged branch to the current version of the network-manager package in the Ubuntu 17.04 repositories! See https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1624317/comments/81
Thanks so much for your work :)

[Boris Malkov (hikari968)]

It already looks like some kind of a tradition for ubuntu to break something critical in every single release and keep those bugs as long as possible.

[demizer (jeezusjr)]

Trying the above fix does not work for 17.10. This is highly unfortunate.

[Ricardo (nu118yt3)]

This bug still exists in Ubuntu 17.10 (ProtonVPN). In distros as Arch, Manjaro and Fedora it never happened. Is this going to 18.04 LTS as well?! Why no one cares?

[Sebastien Bacher (seb128)]

The corresponding GNOME bug has been marked fixed in
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=02d56ec87

that commit is in the 17.10 version, if that doesn't work then it's another issue or the upstream report should be reopened

[Nicholas Stommel (nstommel)]

I'm not sure about split-horizon DNS, frankly I think that is a different bug entirely. However, I have had no problems with DNS leaks over my VPN connections whatsoever on Ubuntu 17.10. The bugfix I personally requested from the NM-devs and backported to Ubuntu 17.04 (running NetworkManager v1.4.x) was effectively just patching the negative dns-priority bug related to systemd-resolved. From 'man nm-settings': "Negative values have the special effect of excluding other configurations with a greater priority value; so in presence of at least a negative priority, only DNS servers from connections with the lowest priority value will be used." This means that DNS servers configured for the non-VPN connection will be 'unseated' and ONLY the VPN-configured DNS servers are used.

Ubuntu 17.10 is running NetworkManager v1.8.4, so Thomas Haller's merged bugfix is present and working. You MUST use the command:

'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'

or similar to actually set negative DNS priority for the VPN connection. Restart the network manager with 'sudo service network-manager restart', then connect to the VPN. Examine the output of 'systemd-resolved --status' and use the 'Extended' test on dnsleaktest.com to verify that you are not leaking DNS queries. I use openvpn, but setting negative dns priority should work for preventing DNS leaks over regular VPN connections of all kinds as a kind of 'catch-all'.

Auto-connecting to openvpn through the GUI is a little troublesome in 17.10, but this 'fix' worked for me:
https://askubuntu.com/questions/967408/how-to-automatically-connect-to-vpn-in-ubuntu-17-10/967415#967415

[Ricardo (nu118yt3)]

#103 did fix it

Adding:

[ipv4]
dns-priority=-42

to system-connections config file
or runing 'sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42'
and restarting networkmanager service
did fix dns leaking using ProtonVPN on openvpn for me, thanks.

But i didn't quite understand the problem! Is it the provided openvpn config file misconfigured or a networkmanager bug? As i said, in others distros doesn't happend!

[Thomas (tombl)]

The issue I had either was this bug or something else, but somehow it's apparently working in 17.10. Basically, the domain(s) of the corporate vpn that I connect to resolve over the VPN again, while everything else resolves as usual. This worked fine in 16.10, was entirely broken in 17.04 and is back to working in 17.10.

[Jordi Miralles (jmiralles)]

Confirming is working again in 17.10

[Christian Ehrhardt  (paelzer)]

Per former comments setting 17.10 to fix released.

[bagl0312 (bagl0312)]

Hi,
I confirm that with the command:

sudo nmcli connection modify <vpn-connection-name> ipv4.dns-priority -42

there is not anymore DNS leakage.

However I am wondering why this command is needed, why the fix released cannot include it by default ?

[Nicholas Stommel (nstommel)]

@bagl0312 I agree, there really should be some kind of GUI default way to set negative DNS priority when setting up certain VPN connections. The average user shouldn't experience a nasty surprise when DNS leaks happen by default.

[flux242 (flux242)]

I'm not sure if setting negative priority really solves the dns leaks problem because I'm on 17.10 and I do have dns leaks. If I'm connected to my ISP over a LTE network and the connection is unstable then it could happen that DNS queries will be sent over my ISP network and not over my VPN connection. The only solution that works for me currently is
sudo systemctl disable systemd-resolved.service
sudo service systemd-resolved stop

Put the following line in the [main] section of your /etc/NetworkManager/NetworkManager.conf:
dns=default

Delete the symlink /etc/resolv.conf
rm /etc/resolv.conf

Restart network-manager
sudo service network-manager restart

Caution! Be aware that disabling systemd-resolvd might break name resolution in VPN for some users - according to the original thread https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu