resolvconf lists nameserver addresses in the wrong order

[周成瑞]

Binary package hint: network-manager

I am using Network Manager to connect to my OpenVPN server. When I use the openvpn command line client, DNS push is fine, resolv.conf is updated to the pushed DNS servers (old lines gone). When I use NM, resolv.conf is not written corretly. The OpenVPN server pushes 4 DNS server, but only the first one is appended to resolv.conf. It should remove the old lines and add the 4 servers from the OpenVPN server to resolv.conf.

My daemon.log after connecting:

Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> Starting VPN service 'org.freedesktop.NetworkManager.openvpn'...
Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> VPN service 'org.freedesktop.NetworkManager.openvpn' started (org.freedesktop.NetworkManager.openvpn), PID 5898
Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> VPN service 'org.freedesktop.NetworkManager.openvpn' appeared, activating connections
Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> VPN plugin state changed: 1
Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> VPN plugin state changed: 3
Sep 29 18:26:13 pan-desktop NetworkManager[720]: <info> VPN connection 'VPN 连接 1' (Connect) reply received.
Sep 29 18:26:13 pan-desktop nm-openvpn[5901]: OpenVPN 2.1.0 i686-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Jul 12 2010
Sep 29 18:26:13 pan-desktop nm-openvpn[5901]: WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Sep 29 18:26:13 pan-desktop nm-openvpn[5901]: NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Sep 29 18:26:13 pan-desktop nm-openvpn[5901]: /usr/bin/openssl-vulnkey -q -b 1024 -m <modulus omitted>
Sep 29 18:26:14 pan-desktop nm-openvpn[5901]: LZO compression initialized
Sep 29 18:26:14 pan-desktop nm-openvpn[5901]: UDPv4 link local: [undef]
Sep 29 18:26:14 pan-desktop nm-openvpn[5901]: UDPv4 link remote: [AF_INET]<server_ip>:1194
Sep 29 18:26:18 pan-desktop nm-openvpn[5901]: [server] Peer Connection Initiated with [AF_INET]<server_ip>:1194
Sep 29 18:26:20 pan-desktop modem-manager: (net/tun0): could not get port's parent device
Sep 29 18:26:20 pan-desktop NetworkManager[720]: SCPlugin-Ifupdown: devices added (path: /sys/devices/virtual/net/tun0, iface: tun0)
Sep 29 18:26:20 pan-desktop NetworkManager[720]: SCPlugin-Ifupdown: device added (path: /sys/devices/virtual/net/tun0, iface: tun0): no ifupdown configuration found.
Sep 29 18:26:21 pan-desktop nm-openvpn[5901]: TUN/TAP device tun0 opened
Sep 29 18:26:21 pan-desktop nm-openvpn[5901]: /sbin/ifconfig tun0 10.8.0.6 pointopoint 10.8.0.5 mtu 1500
Sep 29 18:26:21 pan-desktop nm-openvpn[5901]: /usr/lib/network-manager-openvpn/nm-openvpn-service-openvpn-helper tun0 1500 1542 10.8.0.6 10.8.0.5 init
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> VPN connection 'VPN 连接 1' (IP Config Get) reply received.
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> VPN Gateway: <server_ip>
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal Gateway: 10.8.0.5
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Tunnel Device: tun0
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 Address: 10.8.0.6
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 Prefix: 32
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 Point-to-Point Address: 10.8.0.5
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Maximum Segment Size (MSS): 0
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Static Route: 10.8.0.1/32 Next Hop: 10.8.0.1
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 DNS: 4.2.2.1
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 DNS: 4.2.2.2
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 DNS: 4.2.2.3
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> Internal IP4 DNS: 208.67.220.220
Sep 29 18:26:21 pan-desktop NetworkManager[720]: <info> DNS Domain: '(none)'
Sep 29 18:26:21 pan-desktop nm-openvpn[5901]: Initialization Sequence Completed
Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> (tun0): writing resolv.conf to /sbin/resolvconf
Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> VPN connection 'VPN 连接 1' (IP Config Get) complete.
Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> (tun0): writing resolv.conf to /sbin/resolvconf
Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> Policy set 'VPN 连接 1' (tun0) as default for IPv4 routing and DNS.
Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> VPN plugin state changed: 4
Sep 29 18:26:22 pan-desktop nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/01ifupdown' exited with error status 1.

My daemon.log after disconnecting:

Sep 29 18:28:14 pan-desktop nm-openvpn[5901]: /sbin/ifconfig tun0 0.0.0.0
Sep 29 18:28:14 pan-desktop NetworkManager[720]: <info> (tun0): writing resolv.conf to /sbin/resolvconf
Sep 29 18:28:14 pan-desktop avahi-daemon[726]: Withdrawing workstation service for tun0.
Sep 29 18:28:14 pan-desktop nm-openvpn[5901]: SIGTERM[hard,] received, process exiting
Sep 29 18:28:16 pan-desktop NetworkManager[720]: <info> (ppp0): writing resolv.conf to /sbin/resolvconf
Sep 29 18:28:16 pan-desktop NetworkManager[720]: <info> Policy set 'DSL 连接 1' (ppp0) as default for IPv4 routing and DNS.
Sep 29 18:28:16 pan-desktop NetworkManager[720]: SCPlugin-Ifupdown: devices removed (path: /sys/devices/virtual/net/tun0, iface: tun0)
Sep 29 18:28:16 pan-desktop nm-dispatcher.action: Script '/etc/NetworkManager/dispatcher.d/01ifupdown' exited with error status 1.

I am using Maverick beta, NM 0.8.1. I think the problem is at this file http://is.gd/fzBN8.

[周成瑞 (e93b5ae3)]

My resolv.conf before VPN connection:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 221.7.34.10
nameserver 221.7.34.11

My resolv.conf after VPN connection:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 221.7.34.10
nameserver 221.7.34.11
nameserver 4.2.2.1

[Mathieu Trudel-Lapierre (cyphermox)]

This looks to me like it is pretty much expected behaviour... There couldn't be more namservers appended to the list in resolv.conf as only the first three will ever be used by the libc resolver.

On the other hand, you may also mean that all DNS nameservers received from the VPN should be authoritative and used *instead* of the ones from the local network. In this case, could you confirm that the "Use this connection only for the resources on its network" checkbox, under IPv4 Preferences for the VPN connection is unchecked? Thanks!

[Mathieu Trudel-Lapierre (cyphermox)]

Marking Low, Incomplete, also reassigning to network-manager-openvpn since this is what will be handling the nameservers received from openvpn.

[周成瑞 (e93b5ae3)]

@Mathieu,

"Use this connection only for the resources on its network" checkbox is unchecked

I use VPN to bypass the Great Firewall of China by routing Internet traffic through the VPN. The two original DNS is provided by my ISP, they poisoned youtube, facebook and twitter (at least these three) DNS records. I have to use the pushed DNS and remove the old ones.

If I use the openvpn command line client with the same client config file, My resolv.conf after VPN connection is:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 4.2.2.1
nameserver 4.2.2.2
nameserver 4.2.2.3

This is handled by /etc/openvpn/update-resolv-conf shell script distributed in the openvpn-client package. It seems it handles only 3 of 4 pushed DNS server, but it correctly replaced my DNS as configured and desired.

[周成瑞 (e93b5ae3)]

Befor filing this bug, I read daemon.log, and the line below makes me think network manager did the DNS stuff:

Sep 29 18:26:22 pan-desktop NetworkManager[720]: <info> (tun0): writing resolv.conf to /sbin/resolvconf

If it's done by openvpn plugin, it should have nm-openvpn process name and id like the line below
:

Sep 29 18:26:21 pan-desktop nm-openvpn[5901]: Initialization Sequence Completed

And I find the C source file in nm repository at http://is.gd/fzBN8, though I think this stuff should really be done by the openvpn plugin.

[周成瑞 (e93b5ae3)]

When connecting with PPTP via nm, I get similar results. So I do not think this is about the OpenVPN plugin.

[Mathieu Trudel-Lapierre (cyphermox)]

Trying to reproduce this with the pptp plugin, but I can't reproduce the issue... I will still try with openvpn though.

To be sure, do you have resolvconf running at the time you set the connection? The comments in your resolv.conf indicate you might. You can verify this by checking if /etc/resolv.conf is a symlink to /var/run/resolvconf/run/resolv.conf.

Is your connection set to Automatic under the IPv4 tab? This is what I set and just get the VPN DNS servers *above* the one from my wifi connection, which is, IIUC, what you want to happen?

[周成瑞 (e93b5ae3)]

My /etc/resolv.conf is a symlink to /etc/resolvconf/run/resolv.conf.

These days I find the quirks. My computer is directly connected to an ADSL modem using DSL in nm. In this scenario, I get the VPN DNS servers below the ones from my ISP. Since the ISP provides two servers, I only get one DNS from my VPN appended to them. I also confirmed this using PPTP VPN in nm.

Then I get a home router. I connect the router to the modem to dial up. My computer is behind the router. In this scenario, I get the VPN DNS servers above the ones from my router. Like below for PPTP:

nameserver 8.8.8.8
nameserver 8.8.4.4
nameserver 192.168.1.1

If VPN gives 3 or more DNS servers, then the one from the router will be pushed out.

It seems nm does the DNS stuff different way it the above scenarios. I tested that the openvpn command client prepends the DNS servers from the VPN. So I wonder if nm will do this the same way, preferably prepend them?

[Pentarh Udi (pentarh)]

I found a bug related to this one.

NetworkManager must REWRITE resolv.conf with new DNS servers obtained from openvpn, rather appending these ones to resolv.conf.

I.e. if NM obtained DNS servers 1.1.1.1 and 2.2.2.2 from openvpn, resolv.conf must contain ONLY them after success connection, because previous DNS servers may become unreachable.

In my case resolv.conf BEFORE openvpn connection is:
---------
nameserver 212.48.193.37
nameserver 192.168.100.1
---------

And after is:
---------
# Generated by NetworkManager
nameserver 88.85.66.222
nameserver 78.140.128.205
nameserver 213.158.7.2
# NOTE: the libc resolver may not support more than 3 nameservers.
# The nameservers listed below may not be recognized.
nameserver 212.48.193.37
nameserver 192.168.100.1
--------

In this case last three servers are invalid as they are not reachable after VPN connection, so name resolve becomes totally slow after openvpn connection because libc tries to get DNS answer from all servers:

--------------

root@pentarh-netbook:/var/log# tcpdump -i tun0 -n port 53
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type RAW (Raw IP), capture size 65535 bytes
22:33:46.803557 IP 10.20.10.6.55426 > 213.158.7.2.53: 32890+ A? mail.google.com. (33)
22:33:51.807076 IP 10.20.10.6.58861 > 212.48.193.37.53: 32890+ A? mail.google.com. (33)
22:33:55.521957 IP 10.20.10.6.60601 > 213.158.7.2.53: 49670+ A? www.google.com. (32)
22:34:00.527135 IP 10.20.10.6.57982 > 212.48.193.37.53: 49670+ A? www.google.com. (32)
22:34:09.760264 IP 10.20.10.6.39286 > 88.85.66.222.53: 27804+ A? pagead2.googleadservices.com. (46)
22:34:09.946468 IP 88.85.66.222.53 > 10.20.10.6.39286: 27804 5/4/4 CNAME pagead.l.google.com., A 209.85.149.167, A 209.85.149.164, A 209.85.149.165, A 209.85.149.166 (276)
22:34:11.505444 IP 10.20.10.6.45653 > 213.158.7.2.53: 41142+ A? chatenabled.mail.google.com. (45)
--------------

As you can see, libc tries to resolve mail.google.com from old unreachable servers and gets the answer from correct DNS after 20 seconds (!!!) of first query.

This should be fixed, it makes OpenVPN plugin for NM unusable.

[Pentarh Udi (pentarh)]

And somebody, update pls proirity for this bug to higher.

[Mathieu Trudel-Lapierre (cyphermox)]

Sadly, there is currently no true consent on which DNS servers should be used (though I agree defaulting to the ones from VPN often makes the most sense). Please, bring this up on the NetworkManager mailing list (http://mail.gnome.org/mailman/listinfo/networkmanager-list) for how to deal with the various cases (having DNS from VPN appended or prepended to the existing DNS servers in /etc/resolv.conf).

As a workaround, you should be able to use resolvconf which is recommended and usually installed alongside openvpn and other VPN plugins and configure it to do exactly what you need (be it ignore the vpn DNS settings from NM, or keep those above, whichever).

[Pentarh Udi (pentarh)]

>> how to deal with the various cases (having DNS from VPN appended or prepended to the existing DNS servers in /etc/resolv.conf).

They should neither be appended nor prepended. Because in any case libc uses all of them.

DNS from VPN must be exclusive in resolv.conf while VPN connection is active in a case if VPN peer provided any DNS.

pure openvpn binary do this correctly - it makes obtained DNS servers exclusive in resolv.conf, and when VPN connection closed, it restores original DNS servers.

I think NM should follow this functionality.

[MarcH (marc-h38)]

> Because in any case libc uses all of them.

This does not seem correct. If I run "getent ahostsv4 www.google.com" AND the first nameserver answers, then other nameservers are never queried.

[MarcH (marc-h38)]

> DNS from VPN must be exclusive in resolv.conf while VPN connection is active in a case if VPN peer provided any DNS.

Not really. In many cases it is best to still use the original DNS servers since they are much closer.

It depends.

[MarcH (marc-h38)]

It looks like this debian patch could help:
 http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=272265 (glibc rereads resolv.conf)

Doesn't Ubuntu have it?

[周成瑞 (e93b5ae3)]

2011/1/12 MarcH <email address hidden>

> > DNS from VPN must be exclusive in resolv.conf while VPN connection is
> active in a case if VPN peer provided any DNS.
>
> Not really. In many cases it is best to still use the original DNS
> servers since they are much closer.
>
> Then what is the meaning of VPN DNS? If VPN sends DNS servers, nm should
use it as the OpenVPN official client does.

I hate it when I was first reporting this bug someone arrogant quickly
ingored it.

> It depends.
>
> --
> You received this bug notification because you are a direct subscriber
> of the bug.
> https://bugs.launchpad.net/bugs/651007
>
> Title:
> Does not correctly write resolv.conf
>

[MarcH (marc-h38)]

"use the DNS servers from the VPN" does NOT imply: "ignore the local DNS servers". Why would it? In *your* case the local DNS servers become unreachable, but not in mine. So in my case it is the command line client which is "buggy"

It is expected for some DNS servers to be not responsive: this is exactly why several of them should be configured.

This more and more looks like a glibc problem, not really related to NM. It is hard to ask NM to try to work around every limitation of the resolver in glibc. Suppose for a second that the resolver in glibc were to:
- re-read any change to resolv.conf immediately
- support more than 3 servers
- query all of them simultaneously and return as soon as it gets an answer
... then would you still have a problem? Looks like not!

If you find glibc resolver too limited, then try dnsmasq or an alternative resolver as you were suggested?

[Mathieu Trudel-Lapierre (cyphermox)]

MarcH, I'm not sure if it's an issue in eglibc, but from what I can tell the patch you mentioned has been in Ubuntu for a long while (and is actually in the upstream eglibc release at this point).

Pentar, Pan,
NM requires having all the DNS servers available for cases such as split-tunnelling in VPNs. As such, such a functionality will probably not be removed. However, since it seems like the order in which the DNS nameservers are being written is not consistent, the issue you are reporting is an upstream one and it would be nice if somebody could send the bug to the developers of NetworkManager by following the instructions at https://wiki.ubuntu.com/Bugs/Upstream/GNOME. If you have done so, please tell us the number of the upstream bug (or the link), so we can add a bugwatch that will inform us about its status. Thanks in advance.

[周成瑞 (e93b5ae3)]

I have reported this at GNOME bugzilla here: https://bugzilla.gnome.org/show_bug.cgi?id=641593.

[周成瑞 (e93b5ae3)]

Hi all,

This is caused by resolvconf. If I uninstall resolvconf, NetworkManager will put the DNS servers from VPN at the beginning of resolv.conf, which in expected. This bug has been resolved at the GNOME Bugzilla.

[Thomas Hood (jdthood)]

@Yongzhi Pan: Let's get to the bottom of this. Please install resolvconf, reboot, connect to VPN, check that /etc/resolv.conf is wrong. Then send the output of

ls -l /etc/resolv.conf
cat /etc/resolv.conf
ls -l /run/resolvconf
ls -l /run/resolvconf/interface
for F in /run/resolvconf/interface/* ; do echo === $F === ; cat $F ; done
ls -l /etc/resolvconf/resolv.conf.d
for F in /etc/resolvconf/resolv.conf.d/* ; do echo === $F === ; cat $F ; done

and also say what you think the contents of /etc/resolv.conf should be.

[Steve Langasek (vorlon)]

All the bug activity here predates the rework of resolvconf integration in precise. Is there any reason to think there's still a bug at all in later releases?

[Thomas Hood (jdthood)]

Hi Steve.

The complaint here is that with resolvconf installed, the nameservers are ordered differently than without --- and incorrectly. More specifically, whereas VPN nameservers get listed first by NM when it writes resolv.conf directly, the VPN nameservers are listed last when resolvconf intermediates.

I initially thought that this was #994575, but no.

 I don't think that the relevant aspects of resolvconf were changed during the integration process, so if there was a bug before then it may be still present. But of course it could be a local configuration issue.

[Thomas Hood (jdthood)]

@Yongzhi Pan: Can you please provide us with more information about this problem?

Please install resolvconf, reboot, connect to VPN, check that /etc/resolv.conf is wrong, as you reported in Launchpad bug #651007. Then send the output of

ls -l /etc/resolv.conf
cat /etc/resolv.conf
ls -l /run/resolvconf
ls -l /run/resolvconf/interface
for F in /run/resolvconf/interface/* ; do echo === $F === ; cat $F ; done
ls -l /etc/resolvconf/resolv.conf.d
for F in /etc/resolvconf/resolv.conf.d/* ; do echo === $F === ; cat $F ; done
ls -l /etc/resolvconf/interface-order
cat /etc/resolvconf/interface-order

and also say what you think the contents of /etc/resolv.conf should be.

[Thomas Hood (jdthood)]

No reply from Yongzhi, so closing.