DNS leak in ubuntu 16.10

[Eylul]

(splitting off from https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1634689)

When using a couple of example setup files (different servers) that worked without leak in 16.04, in 16.10 there is dns leak. I am not sure how to exactly send requests mean for VPN but would be willing to try if I can figure it out. The rest of the information is below.

How is the VPN configured on the client?

Export of VPN settings as .ovpn:
client
remote <server> <port>
ca <ca cert>
cert <cert>
key <key>
comp-lzo yes
dev tun
proto udp
nobind
auth-nocache
script-security 2
persist-key
persist-tun
user nm-openvpn
group nm-openvpn

vpn conf at /etc/NetworkManager:

[connection]
id=<id>
uuid=<uuid>
type=vpn
permissions=
secondaries=
timestamp=<timestamp>

[vpn]
connection-type=tls
remote=<IP>:<port>
comp-lzo=yes
cert-pass-flags=0
cert=<cert>
dev=tun
key=<key>
ca=<ca>
service-type=org.freedesktop.NetworkManager.openvpn

[ipv4]
dns-search=
method=auto

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
ip6-privacy=0
method=auto

Is it set up to "Use this connection only for the resources on its network"? Is that the case for both IPv4 and IPv6?
No, and no.

What are the contents of /etc/resolv.conf?

/etc/resolve.conf while VPN is running:
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search home

[Seth Arnold (seth-arnold)]

Could you investigate a bit if this may be related to the new systemd-reosolvd service? What's /etc/nsswitch.conf say for 'hosts'?

Thanks

[Eylul (eylul)]

Hi, here is what the hosts line of /etc/nsswitch.conf is saying, while the leak is occuring

hosts: files mdns4_minimal [NOTFOUND=return] resolve [!UNAVAIL=return] dns myhostname

Thanks. :)

[Eylul (eylul)]

Could you investigate a bit if this may be related to the new systemd-reosolvd service?
*is not sure exactly how to go about testing this*

[Seth Arnold (seth-arnold)]

Eylul, try removing this part: resolve [!UNAVAIL=return]

Thanks

[Eylul (eylul)]

Hi Seth,
1) /etc/nsswitch.conf hosts line now reads:
   hosts: files mdns4_minimal [NOTFOUND=return] dns
2) rebooted
3) vpn on boot: no leak. (I think same as before)
4) I turn off vpn
5) use a browser
6) close browser
7) start vpn connection
8) open browser again

Still leaking. :(

Thanks for looking into this.

[Mathieu Trudel-Lapierre (cyphermox)]

Saying "there's a leak" doesn't mean anything. What makes you think it's the case? What are you looking at to tell it's the case?

Please show the data you're using, preferably with something that looks like an IP address (even if it's not the exact thing in your configuration, it should make sense -- we need to see that the value is the same as what is in logs, etc.)

Additionally, please make sure you include the logs at least from NetworkManager (/var/log/syslog), preferably the whole syslog as things like what dnsmasq says are also relevant.

Also, please try to kill -USR1 the dnsmasq process:

sudo pkill -USR1 -c -e dnsmasq

Then include here what gets written by dnsmasq to /var/log/syslog, both before connecting to the VPN, and after connecting to it. We need to see what nameservers are used on and off the VPN to be able to tell at all if some requests are sent to the wrong server.

[thegreathemant (hemant-khatod)]

I went to ipleak.net to test if my ip was leaking and sure enough, I saw my DNS ip address. Its blocking all other IP address except DNS.

IP Details of 192.168.1.2

[Joe Liau (joe)]

This is still happening in 17.04

[GammaPoint (brad-malone)]

I am seeing DNS leaks in 17.04. I had been running 16.10 and the dnsmasq fix that was released fixed my issue back then. But in Zesty I'm seeing this problem too and not sure how to resolve it yet.

[Seth Arnold (seth-arnold)]

Hello GammaPoint, if your system was working in 16.10 but fails in 17.04 then it'd probably be better for everyone involved if you filed a new bug report with your information. I suggest trying to answer Mathieu's questions from this bug directly in the description of the new bug.

Thanks

[GammaPoint (brad-malone)]

Thank you, Seth. I've attempted to begin the conversation here: https://bugs.launchpad.net/ubuntu/+source/openvpn/+bug/1685391

[Jordi Miralles (jmiralles)]

Hi, I have been experiencing this bug with the nm-openvpn applet and more rarely with the openvpn cli client. Right now I'm havinh a hard time reproducing it but it still definitely happens. I'kk try to get my network to the original state before trying to mitigate it because right now when www.dnsleaktest.com detects my ISP DNS chromium it just freezes the page with these errors:

Failed to load resource: net::ERR_NAME_NOT_RESOLVED
op3bcnhs63.dnsleaktest.com/ Failed to load resource: net::ERR_NAME_NOT_RESOLVED
eti6s7oq49.dnsleaktest.com/ Failed to load resource: net::ERR_NAME_NOT_RESOLVED
4yvhbjv7qc.dnsleaktest.com/ Failed to load resource: net::ERR_NAME_NOT_RESOLVED
u2rotxbca3.dnsleaktest.com/ Failed to load resource: net::ERR_NAME_NOT_RESOLVED
51cy8y0t8y.dnsleaktest.com/ Failed to load resource: net::ERR_NAME_NOT_RESOLVED

When I have full logs about what is happening I will post them.

[Jordi Miralles (jmiralles)]

So, as promised, the logs. The only thing I needed to replicate the issue was to add again a DNS server on the network manager configuration. I found out that removing it from there + using UFW was doing the trick (meaning the test didn't crash when tried to resolve using the alternative DNS server and the possible rogue requests are stopped).

For this I'm using openvpn on the CLI. The issue was more or less the same for the openvpn-nm applet but I wasn't able to find a way to get it to work there.

For connecting I use ovpn files with these options:

client
dev tun
proto udp
remote us-ga.gw.ivpn.net 2049
auth-user-pass /home/tux/pass.txt

resolv-retry infinite
nobind
persist-tun
persist-key
persist-remote-ip

cipher AES-256-CBC
tls-cipher TLS-DHE-RSA-WITH-AES-256-CBC-SHA:TLS-DHE-DSS-WITH-AES-256-CBC-SHA:TLS-RSA-WITH-AES-256-CBC-SHA
ns-cert-type server
verify-x509-name us-ga name-prefix
key-direction 1
comp-lzo
verb 3

;ca ca.crt
<ca>
-----BEGIN CERTIFICATE-----
(...)

</ca>
<tls-auth>
-----BEGIN OpenVPN Static key V1-----
(...)
-----END OpenVPN Static key V1-----
</tls-auth>

up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf
script-security 2

[Jordi Miralles (jmiralles)]

Hi,

I think I found the reason and the solution for this. I left all the logs on stdout for a while to see if anything dodgy appeared and at some point saw this message:

./syslog:May 7 14:16:08 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 14:28:31 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 14:30:05 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 14:37:58 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 14:39:52 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 14:41:34 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 15:29:40 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 15:29:40 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.4.4.
./syslog:May 7 15:31:31 tuxedo systemd-resolved[1434]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 15:45:41 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 15:45:41 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.4.4.
./syslog:May 7 15:49:27 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 15:51:12 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 16:08:55 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 16:08:55 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.4.4.
./syslog:May 7 16:09:10 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 16:09:11 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.4.4.
./syslog:May 7 18:00:55 tuxedo systemd-resolved[1469]: Switching to fallback DNS server 8.8.8.8.
./syslog:May 7 18:33:53 tuxedo systemd-resolved[1435]: Switching to fallback DNS server 8.8.8.8.

Which mas making no sense at all, as those DNS were changed for the DNSWatch ones months ago (and before starting to use a VPN at all, and are nowhere on my config files), I have no DNS servers configured aside of the ones the VPN pushes when I connect.

So I tried this:

sudo systemd-resolve --flush-caches

And ran an extended dns leak test on https://dnsleaktest.com/ while connected both on the openvpn CLI and with the openvpn-network-manager-gnome applet. Both were negative (no leaks) while they used to fail before.

Can someone please test this too and comment?

Thanks,

J

[Jordi Miralles (jmiralles)]

Hi,

I actually needed another thing too for getting rid of all the references to the 8.8.8.8 DNS:

First:

tux@tuxedo:/var/log$ ss -psaux | grep 8.8.8.
u_str ESTAB 0 0 @/tmp/.X11-unix/X0 285858 * 284387
u_str ESTAB 0 0 * 284387 * 285858 users:(("iridium-browser",pid=20582,fd=98))

Action:

tux@tuxedo:/var/log$ dbus-cleanup-sockets

Result:

tux@tuxedo:/var/log$ ss -psaux | grep 8.8.8.
tux@tuxedo:/var/log$

I think this is related to that portion of code (but can't tell for sure, will need some comment here):

https://bazaar.launchpad.net/~ubuntu-branches/ubuntu/vivid/openvpn/vivid/view/head:/src/openvpn/socket.c#L1473

[Jordi Miralles (jmiralles)]

And finally, it seems there was already some work in progress for this in one of the branches on openvpn github up until recently:

https://github.com/OpenVPN/openvpn/blob/release/2.4/src/openvpn/block_dns.c
Branch: Release/2.4

[Jordi Miralles (jmiralles)]

I have kept testing this and the issue has been resolved. Now it should be matter of integrating this on a branch? As I pointed out on #17 the OpenVPN dev team has already a branch that should solve this. Let me know if I'm wrong.

[Jordi Miralles (jmiralles)]

And still adding further input (in hopes it's useful) when this time I tested activating the UFW and the CLI client while the network-manager-openvpn applet was still ON the rogue DNS server appears once again. Keep in mind that this shouldn't really be on any of the configuration files at all. Before testing I had designated 84.200.69.80 as the only resolver for that connection on network-manager.

More logs (syslog, Ununtu 17.04 - 4.10.0-20-generic, all packages up to date) :

Everything was good until I put up the firewall (blocking the VPN DNS on pursose, just to see how it reacted to a stress test)

May 8 04:10:17 tuxedo kernel: [ 2919.884244] [UFW BLOCK] IN= OUT=tun1 SRC=10.43.16.23 DST=10.43.16.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=58632 DF PROTO=UDP SPT=48934 DPT=53 LEN=42
May 8 04:10:17 tuxedo kernel: [ 2919.884259] [UFW BLOCK] IN= OUT=tun1 SRC=10.43.16.23 DST=10.43.16.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=58633 DF PROTO=UDP SPT=48934 DPT=53 LEN=42
May 8 04:10:17 tuxedo kernel: [ 2919.884273] [UFW BLOCK] IN= OUT=tun1 SRC=10.43.16.23 DST=10.43.16.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=58634 DF PROTO=UDP SPT=48934 DPT=53 LEN=42
May 8 04:10:17 tuxedo kernel: [ 2919.884287] [UFW BLOCK] IN= OUT=tun1 SRC=10.43.16.23 DST=10.43.16.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=58635 DF PROTO=UDP SPT=48934 DPT=53 LEN=42
May 8 04:10:17 tuxedo kernel: [ 2919.884302] [UFW BLOCK] IN= OUT=tun1 SRC=10.43.16.23 DST=10.43.16.1 LEN=62 TOS=0x00 PREC=0x00 TTL=64 ID=58636 DF PROTO=UDP SPT=48934 DPT=53 LEN=42
May 8 04:10:17 tuxedo compiz[2489]: WARN 2017-05-08 04:10:17 unity.dash.view DashView.cpp:1272 Search failed 'fire'=> Timeout was reached
May 8 04:10:17 tuxedo unity-scope-hom[5319]: scope.vala:669: Unable to search scope: Timeout was reached
May 8 04:10:17 tuxedo unity-scope-hom[5319]: unity-master-scope.vala:114: Unable to search scope: 'Timeout was reached'
May 8 04:10:20 tuxedo unity-panel-ser[2498]: menus_destroyed: assertion 'IS_WINDOW_MENU(wm)' failed
May 8 04:10:37 tuxedo NetworkManager[1315]: <info> [1494209437.6569] devices removed (path: /sys/devices/virtual/net/tun1, iface: tun1)
May 8 04:10:37 tuxedo NetworkManager[1315]: <info> [1494209437.6579] device (tun1): state change: activated -> unmanaged (reason 'unmanaged') [100 10 3]
May 8 04:10:37 tuxedo dbus[1288]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
May 8 04:10:37 tuxedo systemd[1]: Starting Network Manager Script Dispatcher Service...
May 8 04:10:37 tuxedo dbus[1288]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
May 8 04:10:37 tuxedo nm-dispatcher: req:1 'down' [tun1]: new request (2 scripts)
May 8 04:10:37 tuxedo nm-dispatcher: req:1 'down' [tun1]: start running ordered scripts...
May 8 04:10:37 tuxedo FirewallHandler: Saving iptables rules.
May 8 04:10:37 tuxedo nm-dispatcher[9622]: <30>May 8 04:10:37 FirewallHandler: Saving iptables rules.
May 8 04:10:37 tuxedo systemd[1]: Started Network Manager Script Dispatcher Service.
May 8 04:10:44 tuxedo NetworkManager[1315]: <info> [1494209444.6758] audit: op="connection-deactivate" uuid="9fcd6b62-3762-424f-9b2...

[Vincent (dawansv)]

Jordi:

To me it seems that the problem is that the update-resolv-conf script was designed to work with pure dnsmasq and not systemd-resolved (used since 16.10).

I googled around and found somebody has made an update-systemd-resolved script to replace the update-resolv-conf script when systemd-resolved is in play.

See my detail explanations here: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1624317/comments/42

I works well for me at least.

[Nicholas Stommel (nstommel)]

Sadly, the option block-outside-dns is only supported on Windows clients. Which is a real shame, because systemd-resolved is leaking DNS queries everywhere by design. This is a problem with the hardcoded design of the gnome network manager integrating (or rather...not integrating) with systemd-resolved. I attempted to understand GIO proxy bus calls but honestly patching the network manager is beyond my capabilities.