dns server priority broken

[Thomas M Steenholdt]

network-manager: 1.2.4-0ubuntu1

Yakkety appears to have switched back from resolved to dnsmasq, but it seems server priority/order is broken.

Example: In split DNS setups, connecting to VPN will not cause us to query the DNS provided by the VPN first (or only), which should be the proper way to resolve names in that case.

Say server.example.com in the public DNS resolves to a.a.a.a and in the private DNS resolves to b.b.b.b.

Stuff would work from my normal internet-connection, but connection to VPN would cause stuff to misbehave. I expect to hit the b.b.b.b address but since my normal LAN DNS is being used first, I'm really hitting a.a.a.a.

Please let me know how to proceed - Hopefully this can be fixed in time for release.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Lonnie Lee Best (launchpad-startport)]

If the "order in which dns servers are queried" is flawed, as you've discovered. That might be the root cause of a lot of other outstanding bugs:

VPN - "Additional DNS servers" Settings are being Ignored:
https://bugs.launchpad.net/network-manager-applet/+bug/1633874

network-manager ignoring DNS settings in wifi connections:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1629276

VPN - "Additional Search Domains" Settings are being Ignored:
https://bugs.launchpad.net/network-manager-applet/+bug/1633877

[Lonnie Lee Best (launchpad-startport)]

Can someone figure out which of these debian bugs should be linked with this Ubuntu bug report:

https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=dnsmasq;dist=unstable

[Lonnie Lee Best (launchpad-startport)]

Is this the cause:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777004

???

[Thomas M Steenholdt (tmus)]

No, that does not seem like the problem to me - Or it's not described correctly. I can't seem to find a proper match in that list.

resolv.conf is not the issue here - the problem is in the DNS servers that dnsmasq uses. I.e. one is added to dnsmasq when my wireless connection comes up, and when I launch my VPN, to more are APPENDED. And it seems like dnsmasq will try by wifi DNS first, VPN DNS later.

So for split DNS, the VPN DNS servers are not queried at all, causing problems. The proper way is probably to PREPEND the VPN DNS servers to the list of DNS servers that dnsmasq uses or otherwise prioritize these over previously configured servers. That way everything should work as expected.

[Lonnie Lee Best (launchpad-startport)]

One of us is going to have to endure submitting a new bug to debian regarding this issue and then linking that bug to this ubuntu bug. I think their bug system is email based and it is always an extra pain for me to deal with. So, I'd appreciate not having to be the one to do this. Thomas seems to have a good handle on this issue (hint hint).

[Thomas M Steenholdt (tmus)]

I'll do it :)

Any helpful pointers to which package/version i should reference, since I don't have the real debian version of this package anywhere?

[Lonnie Lee Best (launchpad-startport)]

You can find out with this command:
cat /etc/debian_version

Source:
http://askubuntu.com/questions/445487/which-ubuntu-version-is-equivalent-to-debian-squeeze

[Lonnie Lee Best (launchpad-startport)]

For me, on 16.04.1, that's:
debian stretch/sid

I'm experiencing the issue on both 16.04.1 and 16.10. I posted a work around here:
https://bugs.launchpad.net/network-manager-applet/+bug/1633874

[Thomas M Steenholdt (tmus)]

Here's the thing,

I'm on 16.10 which has the debian equiv listed as stretch/sid and my network-manager version is 1.2.4-0ubuntu1.

Looking at Debian stretch and sid, they have network-manager packages in various versions but not 1.2.4.

Furthermore, since Ubuntu employ their own patches, backports, versions, scripts, configuration etc, It'd be wrong of me to file this bug upstream, since I'd certainly risk wasting the Debian folks' time with something that might very well be Ubuntu specific. I simply cannot be certain.

The only proper way to do this, would be for the Ubuntu maintainer to have a look at this bug and decide whether we're dealing with an Ubuntu specific config issue or something upstream and take the proper action upstream if needed.

I have found similar but not identical reports in Debian, so I cannot reference anything that resembles duplicates.

/T

[Bryn Cooke (bryncooke)]

Does anyone know a workaround for 16.10? There is no option to downgrade the package there.

[Mathieu Trudel-Lapierre (cyphermox)]

This isn't how VPNs are supposed to work. I understand that things get incredibly complicated in this case, but they are complicated in the first place.

Some things to watch out for:

 - IPv4 and IPv6 should both be configured the same way; you want either both to be set to split-tunnelling ("Use this connection only for the resources on its network" under Routes), or both not. Having the two use a different setting will make things not work correctly (you will have the wrong set of DNS nameservers configured in dnsmasq, even if you only have IPv4 nameservers).

 - Adding separate nameservers and search domains in the UI may be handled very differently than receiving nameservers from the VPN itself.

If you're seeing this bug, please *file your own* bug report in launchpad, and make sure you add at least the *debug* logs from NetworkManager as well as mentioning exactly how the IPv4/IPv6 and underlying Routes dialogs are configured. To add debug logs for NetworkManager for VPNs, you may run 'nmcli general logging level debug' before reproducing the issue. The information will show up in /var/log/syslog (which is the file you want to include, after reviewing it to remove any sensitive information such as the exact IPs and domain names, or making the bug private).

People should not mark their own bugs as duplicate of others and these kinds of issues are complicated and easily confused as the same thing when they might not be. There is also no point in reporting this on Debian, as they likely don't have the same patches applied.

[Thomas M Steenholdt (tmus)]

Wow - Long message, but what I got from it was "I need to see a debug log", correct? I'll attach that...

I'll also point you to the problematic part:

Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1915] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.48@vpn0' for domain "workdom.lan"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1915] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.48@vpn0' for domain "22.60.10.in-addr.arpa"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1916] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.48@vpn0' for domain "23.60.10.in-addr.arpa"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1916] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.49@vpn0' for domain "workdom.lan"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1916] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.49@vpn0' for domain "22.60.10.in-addr.arpa"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1916] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.49@vpn0' for domain "23.60.10.in-addr.arpa"
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1916] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.48@enp0s31f6'
Dec 5 15:14:48 bar14860 NetworkManager[921]: <debug> [1480961688.1917] dnsmasq[0x560b551920f0]: adding nameserver '10.60.180.49@enp0s31f6'

Now for this test, I'm using the same DNS before and after, but the VPN connection adds DNS resolution using the VPN provided DNS ONLY for the domains it feels is behind the VPN, and there's nothing I can seemingly do to change that behaviour. This is the exact assumption that breaks VPN for many users. When I connect to VPN, especially one that becomes my default gateway, I need all DNS requests to go the the DNS servers behind the VPN.

Need more info, let me know.

[Thomas M Steenholdt (tmus)]

Unfortunately not much traction here, and this appears to annoy people across distros.

In the meantime, an ugly hack is to manually add all internal domains to the NetworkManager VPN config file's dns-search= parameter:

dns-search=domain1.lan;domain2.lan;domain3.lan;example.com;

This causes NetworkManager to split DNS all lookups for these domains to the VPN DNS server, but with the added overhead of searching through all domains for non-existing hostname queries (make sure the primary internal domains are mentioned first). Also, for a multi-city setup like ours, I need to add A LOT of domains to get a functional DNS while on VPN - Including in-addr.arpa specifications for all IP subnets.

So there's a way to sweeten the deal - but this is by no means anything other than a hack.

[Lonnie Lee Best (launchpad-startport)]

Why is the status of this incomplete. I've verified this issue from numerous perspectives that probably all fundamental point directly to the issue Thomas has reported; see post#2.

This is a terrible bug. Network-Manager updates were prematurely released to 14.04 stable and all succeeding releases and no one caught this. I keep waiting as patiently as possible. Please fix!

[Bryn Cooke (bryncooke)]

This workaround worked for me in 16.10:

In /etc/NetworkManager/NetworkManager.conf

Comment out:
dns=dnsmasq

The run:
sudo service NetworkManager restart

[Thomas M Steenholdt (tmus)]

That is certainly a possibility, but unfortunately returns us to a state where applications have to be restarted when nameservers change (glibc resolved.conf issue). That's probably even worse in my situation at least.

[Lonnie Lee Best (launchpad-startport)]

Bryn, thank for the suggested work around. However, it is not a workaround for me. My issue is that I have over 100 Remmina RDP desktop configurations saved that connect to numerous computers on different VPNs by computer name (not by ip address). However, I cannot ping those computers by name when I'm connected to their respective VPNs anymore like I could before recent Network Manger updates. My issue is described in detail here:
https://bugs.launchpad.net/network-manager-applet/+bug/1633874

When you are connected to a VPN, your computer is suppose to check that virtual private network's DNS servers too, when it cannot find (on your local network) the computer-name you are you are trying to reach by name.

Network Manger gives you the opportunity to specify "Additional DNS Servers" for each connection you set up whether it be LAN, WiFi, or VPN. Yet Network manager now fails to queries those DNS servers in the proper order. It seems to just give up at the local DNS level when the other DNSs you've configured could certainly tell you the ip address of the computer name you are trying to reach if it were simply asked. It is not being ASKED.

[gatopeich (gatoguan-os)]

Downgrading network-manager package (not the VPN ones) to v1.1.93 solves the issue so this is an obvious regression.

Note that there was another bug marked as a duplicate of this one but with a more user-friendly description, along with the workaround.

[gatopeich (gatoguan-os)]

Seems fixed in version 1.2.6

[Thomas M Steenholdt (tmus)]

Seems to work in 1.4.4-1ubuntu3 in 17.04 at least... Thanks

[Lonnie Lee Best (launchpad-startport)]

@gatoguan-os : You must have been speaking about the work-around here?
https://bugs.launchpad.net/network-manager-applet/+bug/1633874

Is the above really a duplicate or not?

Lately, I have not even been able to get that work-around (in the link above) to work in 16.04.

Also, this is a related bug, but not exactly what's reported here (it might be a ramification of what's reported here):
https://bugs.launchpad.net/network-manager-applet/+bug/1633877

Right? I hope all these ramifications get fixes.

@tmus : I wonder if 1.4.4-1ubuntu3 will make it into 16.04 (Long term release).

[Thomas M Steenholdt (tmus)]

This is not fixed, but is marked as a duplicate of #1624317