DNS leak in Xubuntu 17.04

[GammaPoint]

I recently installed Xubuntu 17.04 and am seeing DNS leaks after connecting with my VPN (as seen from www.dnsleaktest.com and similar sites). A couple weeks ago, on 16.04 and 16.10, I had similar issues, but they were fixed with an update to dnsmasq (see https://bugs.launchpad.net/ubuntu/+source/dnsmasq/+bug/1639776). However, that fix doesn't seem to be working for me anymore in 17.04.

I've included my /var/log/syslog, which I hope provides some useful information. Happy to give whatever else is needed.

I see the DNS leaks both when connecting through network-manager (my normal way) as well as using openvpn from the commandline.

[Christian Ehrhardt  (paelzer)]

Hi GammaPoint,
thank you for your report bing split from the already complex old report.
Lets try to get into your issue.

I think I understand that you set up your vpn and resolv in a way that you expect any DNS info to be handled "there" inside your VPN but you now see DNS requests being made out-of-band to that - which means "from your computer to the remote DNS" instead of "from your computer to something in your VPN and from there to the DNS". - That is my understanding of a DNS leak, please confirm or correct.

Could you outline your setup about:
- how the DNS in the VPN is configured to cache/forward
- local resolver configuration /etc/resolvconf/* and /etc/resolv.conf
- any further local dns configuration (dnsmasq, named, whatever you have active)
- I think a "dig +trace <somedomain>" could be useful as well, IIRC we should see there which dns your system is asking.
- If the latter is a local dnsmasq or such try to check who its config is set to forward the request to

[GammaPoint (brad-malone)]

Your description of DNS leak is consistent with my own understanding. Specifically, DNS testing sites show my own ISP being used instead of being that of the VPN.

As for my setup, I simply follow the Private Internet Access Linux step-by-step instructions here: https://www.privateinternetaccess.com/forum/discussion/18003/openvpn-step-by-step-setups-for-various-debian-based-linux-oss-with-videos-ubuntu-mint-debian .

In short, the instructions tell you to:
1). Update/install packages like openvpn, network-manager, etc.
2). Download ovpn and crt files from the PIA website
3). Add them to network-manager and make a couple changes before saving.

This procedure has worked for me in the past (as recently as a few weeks ago on 16.10 after the dnsmasq issue was corrected), but they are not working for me on 17.04. I imagine most of the settings you are interested in are located in the ovpn file that I imported, which I include below as an attachment. Since it's short, I'll also append the file info to the bottom of this message.

Does this answer your questions? If not, I'm happy to add whatever other information you might need. I'm a pedestrian user of VPN, so you might need to be explicit in what command line instructions you need me to execute. Thanks!

P.S. While it's of course possible that I'm just doing something stupid since installing 17.04, here is another recent result I found online where someone is experiencing new issues in 17.04: https://www.privateinternetaccess.com/forum/discussion/23756/pia-client-on-ubuntu-17-04-dns-leak, although they are using the official PIA client, whereas I am not.

----- [ovpn file info]
client
dev tun
proto udp
remote us-east.privateinternetaccess.com 1198
resolv-retry infinite
nobind
persist-key
persist-tun
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass
comp-lzo
verb 1
reneg-sec 0
crl-verify crl.rsa.2048.pem
ca ca.rsa.2048.crt
disable-occ

[GammaPoint (brad-malone)]

As for dig +trace, I just did that on www.ubuntu.com, with the attached output (this is on my VPN).

Note: I'm not sure why this is, but sometimes using the dig +trace command will simply return much less info, like so:

dig +trace www.ubuntu.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> +trace www.ubuntu.com
;; global options: +cmd
;; Received 28 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms

Perhaps this is expected, but I thought I'd mention it.

[Simon Déziel (sdeziel)]

Seeing your /etc/resolv.conf (and the other info Christian asked about) would be quite useful. I noticed that when NM is given multiple DNS resolvers to use for the VPN, it sends the same query in parallel to all of them. It's just a speculation but it could be possible for NM/dnsmasq to also include the other nameservers (those of the ISP) in the parallel resolution?

[Christian Ehrhardt  (paelzer)]

Thanks Simon, the comment on the potential parallel search is great and could be the source of your leak.

From the trace you sent it seems when shrunken down to the path like this:

# you first ask local dnsmask
;; Received 239 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
# that then asks main dns servers
;; Received 866 bytes from 202.12.27.33#53(m.root-servers.net) in 400 ms
;; Received 678 bytes from 192.5.6.30#53(a.gtld-servers.net) in 77 ms
# dns service provider
;; Received 107 bytes from 204.13.251.27#53(ns4.p27.dynect.net) in 197 ms
# canonical name server
;; Received 171 bytes from 91.189.91.139#53(ns3.canonical.com) in 134 ms

But if I understood dig +trace enough it does so by understanding the dns reply.
So your local dnsmasq or such on 127.0.0.1 is reporting "answer from 202.12.27.33#53(m.root-servers.net)" - then it asks this server next which then answers ...

If anything it seems that already your local dns cache/proxy is not asking your "in-vpn" DNS but a public one.

Configs will certainly help a bit in trying to understand that.

[GammaPoint (brad-malone)]

Thanks for the comments. My /etc/resolv.conf is attached. There's a lot in the /etc/resolvconf/ directory -- just let me know if you'd like anything from there and I'll grab it as well.

[GammaPoint (brad-malone)]

Is this discussion relevant to what we're talking about?
https://superuser.com/questions/1153203/ubuntu-17-04-systemd-resolved-dns-lookups-randomly-fail

[Simon Déziel (sdeziel)]

I'm not familiar with systemd-resolved but it seems that you are not
using it as otherwise you'd have 127.0.0.53 in /etc/resolv.conf?

If you could run a packet capture while you do DNS lookups when
connected to the VPN that would be useful. You could capture with:

  sudo tcpdump -w /tmp/dns.pcap -ni any port 53

Then resolve some queries you expect to be answered by your VPN provided
DNSes. After that, please attach the /tmp/dns.pcap in here. FYI, this
will show us what you resolved so don't run the capture while you are
doing anything private.

[GammaPoint (brad-malone)]

Sure thing, Simon. Here is the tcpdump. I then tried to access the https://www.dnsleaktest.com/ site which showed that I was experiencing the DNS leak. If you need anything else (or need that in ASCII), just me know. Thanks for looking into this.

[Simon Déziel (sdeziel)]

Thanks for the pcap. Some extracts with comments:

# queries are apparently sent in parallel
12:31:29.821205 IP 127.0.0.1.58683 > 127.0.0.1.53: 7913+ A? v6r6wsfsgj.dnsleaktest.com. (44)
12:31:29.821307 IP 127.0.0.1.40453 > 127.0.0.53.53: 37214+ A? v6r6wsfsgj.dnsleaktest.com. (44)
12:31:29.821586 IP 127.0.0.1.59554 > 127.0.0.1.53: 40498+ [1au] A? v6r6wsfsgj.dnsleaktest.com. (77)
# 192.168.0.1 is probably your LAN's resolver/ISP provided router (this is the leak)
12:31:29.821655 IP 192.168.0.104.56226 > 192.168.0.1.53: 675+ [1au] A? v6r6wsfsgj.dnsleaktest.com. (77)
# 209.222.18.218 is resolver2.privateinternetaccess.com (what you should be using exclusively to avoid leaks)
12:31:29.821725 IP 10.68.10.6.46982 > 209.222.18.218.53: 8175+ [1au] A? v6r6wsfsgj.dnsleaktest.com. (77)
# responses
12:31:29.865576 IP 209.222.18.218.53 > 10.68.10.6.46982: 8175 NXDomain 0/1/1 (102)
12:31:29.873446 IP 192.168.0.1.53 > 192.168.0.104.56226: 675 NXDomain 0/1/1 (102)

So it looks like systemd-resolved asked the same query roughly simultaneously (70 microsecond interval) to 192.168.0.1 and 209.222.18.218. The systemd-resolved(8) man page explains this:

> Multi-label names are routed to all local interfaces that have a DNS sever configured, plus the
> globally configured DNS server if there is one. [...]
>
> If lookups are routed to multiple interfaces, the first successful response is returned (thus
> effectively merging the lookup zones on all matching interfaces). If the lookup failed on all
> interfaces, the last failing response is returned.

[Christian Ehrhardt  (paelzer)]

Thanks Simon and Gamma for the extra insights!

I don't want to get into politics behind all that but this case appears to be point #8 on this list https://lists.dns-oarc.net/pipermail/dns-operations/2016-June/014964.html

There is this for domain limited networks https://github.com/systemd/systemd/commit/b9fe94cad99968a58e169592d999306fd059eb14 but our case here is about generally not asking "everybody" when dialing up a VPN for privacy.

@Gamma
It should be good to confirm that further by checking the status of it.
$ systemd-resolve --status
I'd expect in your case that this reports two links (local net + vpn) with dns servers each.

You might also test and verify the theory that systemd-resolved's behaviour is the root cause here by switching back to dnsmasq for a try:
https://askubuntu.com/a/899338/532139
If you happen to do so ensuring with a new pcap file would be great.

Once confirmed I thought that we add a bug task for systemd then, but I found a lot of already filed issues around all of this and I think it would be a dup to bug 1624317 then.
There are some suggested workarounds/configs in there as well that you could try for now.

[GammaPoint (brad-malone)]

Thanks for the research on this guys. I had been a idle spectator to the systemd controversies, but didn't realize that I might be bumping up into those choices in a real way myself.

Attached is my systemd-resolve --status. I imagine it shows what you are talking about.

And I also went ahead and tried to switch to dnsmasq and see if that fixed the problem. Unfortunately, it seems that I may need to do something different to actually shut down systemd-resolved. I tried those instructions with a restart, and systemd-resolved was still running. I tried those instructions without a restart, but either systemd-resolved started up again by itself or perhaps by me reconnecting to my VPN via network-manager. I did confirm that systemd-resolved was at some point disabled after executing the commands in step #2 of those instructions, but not sure what started it up again.

Correct me if I'm wrong, but DNS leaking via systemd related issues should be a pretty high priority bug, correct? If it's not the case, and your sense is that there are a lot of technical or political hurdles to this being corrected, perhaps it makes sense for me to return to 16.04 in the meantime.

[Christian Ehrhardt  (paelzer)]

Hi,
yes the output confirms what we thought on DNS per link.
Combined with the concurrent query this creates the DNS Leak you are facing.

I - personally - agree that it is a high priority case.
The reasons for that are that:
- security in general is considered important (unfortunately privacy is only the little brother of security)
- in some sense this is a regression to former behavior which makes it more important

I'd close this bug as a dup to the other one - please weight in there.
For severity/importance as well as eventually when looking to test various situations it is better to have all on one issue.
So please continue the discussion there.

[Vincent (dawansv)]

To force all dns lookups to go only to the link created via openvpn, and not to all links simultaneously, I add to add the following to my config file:

dhcp-option DOMAIN-ROUTE .

Also I am using this script: https://github.com/jonathanio/update-systemd-resolved

See my full explanation here:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317/comments/42

[Nicholas Stommel (nstommel)]

For a Network Manager GUI fix, please see my patch towards the bottom of the bug report https://bugs.launchpad.net/bugs/1624317
No more DNS leaks through the openvpn network-manager gui! Please let me know if this works for you, cheers :)

[Judson W (judderwocky)]

There are a lot of reports on other forums about DNS leaks specific to 17.04.

Once again I upgraded to Ubuntu and have found myself regretting it. There is always some poorly thought out decision or bug that destroys my confidence in this software.