Incorrect default routing after vpnc completes

[janl]

We're using vpnc with a password + one time token at work so I run it from the command line. I've been using it for years, this laptop is probably 2 years old, upgrading ubuntu every half year and I've never had this issue before I upgraded to 15.10.

I've put set +x in the vpnc-script, and I'm also tailing syslog in the same window. Got this trace after being accepted, towards the end:

+ set_default_route
+ /sbin/ip route
+ fix_ip_get_output
+ + grepsed -e ^default
 s/ /\n/g
Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> Device 'tun0' has no connection; scheduling activate_check in 0 seconds.
Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> (tun0): Activation: starting connection 'tun0' (498de0c1-9fe8-43fc-82ac-3d4e1bcbbf2f)
+ sed -ne 1p;/via/{N;p};/dev/{N;p};/src/{N;p};/mtu/{N;p}

HERE the correct default routing is installed:

+ /sbin/ip route replace default dev tun0
+ /sbin/ip route flush cache

And at this point network manager pounces (this is the very next line of the console (from tail -f syslog):

Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> (tun0): device state change: disconnected -> prepare (reason 'none') [30 40 0]
+ [ -n ]
+ [ -n -o -n ]
+ [ -n 194.19.44.43 195.204.29.42 ]
+ modify_resolvconf_manager
+ NEW_RESOLVCONF=
+ NEW_RESOLVCONF=
nameserver 194.19.44.43
+ NEW_RESOLVCONF=
nameserver 194.19.44.43
nameserver 195.204.29.42
+ [ -n ]
Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> (tun0): device state change: prepare -> config (reason 'none') [40 50 0]
+ echo
nameserver 194.19.44.43
nameserver 195.204.29.42
+ /sbin/resolvconf -a tun0

Major networkmanager action:

Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> (tun0): device state change: config -> ip-config (reason 'none') [50 70 0]
Nov 5 11:16:54 niclan-lap NetworkManager[724]: <info> (tun0): device state change: ip-config -> ip-check (reason 'none') [70 80 0]
Nov 5 11:16:55 niclan-lap NetworkManager[724]: <info> (tun0): device state change: ip-check -> secondaries (reason 'none') [80 90 0]
Nov 5 11:16:55 niclan-lap NetworkManager[724]: <info> (tun0): device state change: secondaries -> activated (reason 'none') [90 100 0]
Nov 5 11:16:55 niclan-lap NetworkManager[724]: <info> (tun0): Activation: successful, device activated.
Nov 5 11:16:55 niclan-lap dbus[743]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service'
Nov 5 11:16:55 niclan-lap systemd[1]: Starting Network Manager Script Dispatcher Service...
Nov 5 11:16:55 niclan-lap dbus[743]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Nov 5 11:16:55 niclan-lap systemd[1]: Started Network Manager Script Dispatcher Service.
Nov 5 11:16:55 niclan-lap nm-dispatcher: Dispatching action 'up' for tun0
Nov 5 11:16:55 niclan-lap systemd[1]: Reloading OpenBSD Secure Shell server.
Nov 5 11:16:55 niclan-lap systemd[1]: Reloaded OpenBSD Secure Shell server.
+ run_hooks post-connect
+ HOOK=post-connect
+ [ -d /etc/vpnc/post-connect.d ]
+ exit 0
VPNC started in background (pid: 8778)...
root@niclan-lap:/etc/vpnc#

At this point I cannot reach resources through the VPN.

root@niclan-lap:/etc/vpnc# /sbin/ip route
default via 10.99.64.1 dev wlan0 proto static metric 600
10.21.50.0/24 dev tun0 scope link
10.99.64.0/23 dev wlan0 proto kernel scope link src 10.99.64.195 metric 600
169.254.0.0/16 dev wlan0 scope link metric 1000
193.69.44.30 via 10.99.64.1 dev wlan0 src 10.99.64.195
194.19.44.87 via 10.99.64.1 dev wlan0 proto dhcp metric 600
Nov 5 11:18:51 niclan-lap wpa_supplicant[1358]: nl80211: send_and_recv->nl_recvmsgs failed: -33

As we see the default routing is through the wlan0 instead of tun0. So the default routing set in the vpnc-script is already removed. I can only speculate to _what_ removed it, but network-manager seems to have been active at the time.

So I add the routing again:

root@niclan-lap:/etc/vpnc# /sbin/ip route replace default dev tun0
root@niclan-lap:/etc/vpnc# /sbin/ip route flush cache

root@niclan-lap:/etc/vpnc# /sbin/ip route
default dev tun0 scope link
default via 10.99.64.1 dev wlan0 proto static metric 600
10.21.50.0/24 dev tun0 scope link
10.99.64.0/23 dev wlan0 proto kernel scope link src 10.99.64.195 metric 600
169.254.0.0/16 dev wlan0 scope link metric 1000
193.69.44.30 via 10.99.64.1 dev wlan0 src 10.99.64.195
194.19.44.87 via 10.99.64.1 dev wlan0 proto dhcp metric 600

The routing table is now correct I would say, at least I reach the resources inside the vpn without issue.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: vpnc 0.5.3r550-2
ProcVersionSignature: Ubuntu 4.2.0-16.19-generic 4.2.3
Uname: Linux 4.2.0-16-generic x86_64
ApportVersion: 2.19.1-0ubuntu4
Architecture: amd64
CurrentDesktop: KDE
Date: Thu Nov 5 11:58:20 2015
InstallationDate: Installed on 2013-11-07 (727 days ago)
InstallationMedia: Kubuntu 13.04 "Raring Ringtail" - Release amd64 (20130424)
SourcePackage: vpnc
UpgradeStatus: Upgraded to wily on 2015-10-26 (9 days ago)
modified.conffile..etc.vpnc.default.conf: [deleted]

[Florian Schlichting (fschlich)]

There are several components at play here, apart from the vpnc package there's vpnc-scripts as well as network-manager-vpnc{-gnome}. None of these changed between 15.04 and 15.10. What did change is network-manager itself (from 0.9.10.0-4ubuntu15.1 to 1.0.4-0ubuntu5) and I suspect the network-manager-vpnc package lacks adaptation to handle your case properl. Hence I'm reassigning this bug.

[janl (janl)]

As I said I run vpnc from the commandline due to the need for two factor authentication.

[CParticle (cparticle)]

I'm having a similar issue I just upgrade to 15.10 from 14.04 and I started having the same issue. I would connect from the CLI using vpnc-connect and my saved profile. However once connect I would not be able to access any resource on the other side of the tunnel. It appeared to be a routing error but I had no way of confirming that.

In an effort to solve this I installed nm-applet-vpnc and connected with it and everything worked correctly. I took a look at the routes using netstat -r and there was a sicrepancy.

netstat -r after connect from CLI
$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default 10.0.0.1 0.0.0.0 UG 0 0 0 wlan0
10.0.0.0 * 255.255.255.0 U 0 0 0 wlan0
link-local * 255.255.0.0 U 0 0 0 wlan0
<TUNNEL IP> * 255.255.255.240 U 0 0 0 tun0
<MYPORTAL> 10.0.0.1 255.255.255.255 UGH 0 0 0 wlan0

netstat -r after connect using nm-applet-vpnc
$ netstat -r
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default * 0.0.0.0 U 0 0 0 tun0
default 10.0.0.1 0.0.0.0 UG 0 0 0 wlan0
10.0.0.0 * 255.255.255.0 U 0 0 0 wlan0
link-local * 255.255.0.0 U 0 0 0 wlan0
<TUNNEL IP> * 255.255.255.240 U 0 0 0 tun0
<MYPORTAL> 10.0.0.1 255.255.255.255 UGH 0 0 0 wlan0

I ran
 sudo route add default tun0
Now netstat -r looks correct
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
default * 0.0.0.0 U 0 0 0 tun0
default 10.0.0.1 0.0.0.0 UG 0 0 0 wlan0
10.0.0.0 * 255.255.255.0 U 0 0 0 wlan0
link-local * 255.255.0.0 U 0 0 0 wlan0
<TUNNEL IP> * 255.255.255.240 U 0 0 0 tun0
<MYPORTAL> 10.0.0.1 255.255.255.255 UGH 0 0 0 wlan0

And now everything is working correctly.
However it would be much preferred to not have to manual add the default route once the vpn connection is up.

FYI janl I can get the nm-applet-vpnc to prompt me for dual factor authentication.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[zebul666 (zebul666)]

I am using Ubuntu 15.10 with
 - network-manager 1.0.4-0ubuntu5.1
 - network-manager-openvpn 0.9.10.0-1ubuntu2

And I configured my ethernet connexion to automatically use my openvpn vpn setup when connecting.

If I go to dnsleaktest.com, I found out that networkamanager leaks my dns of my FAI provider and don't use the DNS of the VPN.

However if I close and reopen manually the VPN conenction from networkmanager applet, the DNS leak does not occur anymore.

So the bug occurs only when the VPN connection is set-up automatically when login or coming from sleep state. The DNS are not updated to the ones of the VPN and stays the one previously defined.

[zebul666 (zebul666)]

no. I was wrong to assume it's because the automatic connection.

Once the connection is up, dns could be ok but leaks 5 minutes later withtout the connection ever changing !!!

so wtf !

[zebul666 (zebul666)]

to make things clear, connection uses the LAN DHCP DNS instead of DNS of the VPN tunnel connection setup by network manager

but it seems heratic

[zebul666 (zebul666)]

From the NetworkManager log, it's clear than the previous (before establishing the VPN connection) DNS settings is retained by dnsmasq and not removed.

So you end up with 3 DNS servers settings defined in dnsmasq after the VPN connection is setup, one of which is still the LAN/ISP DNS that causes the random DNS leak. That DNS entry should have deleted.

I don't know if it's upstream bug or ubuntu. I have opened https://bugs.launchpad.net/ubuntu/+source/network-manager-openvpn/+bug/1520771 which shows a work-around for the bug

[Forest (foresto)]

I have observed this problem as well.

Related:
https://bugs.launchpad.net/ubuntu/+source/network-manager/+bug/1211110

[Mincho-gaydarov (mincho-gaydarov)]

I've filled a bug in redhat's bugzilla as I was unable to find this in the beginning. https://bugzilla.redhat.com/show_bug.cgi?id=1322932

I also experience this problem and it is not limited to NetworkManager configuration with dnsmaq. This bug appears also when NetworkManager is managing /etc/resolv.conf directly.

Additional info:
I've tested to deny access to 1-st and 2-nd NS servers from the machine running OpenVPS server and the result was that the local NS server was used, despite the fact that the checkbox 'Use this connection only for resources on its network' is unchecked. This leads to DNS leak if the first 2 NS servers could not be reached.

[Thomas Haller (thaller-1)]

this DNS issue has some similarities with route-management, where users want that once VPN connects no traffic is routed via other devices (bug 749376).

NetworkManager currently doesn't understand a concept that activating a certain connection "vpn0" should prevent DNS/routing via another connection "eth0". For NM, there are just two connections active, both have DNS/routing configured (with differing priorities) and such is the outcome. But the existence/priorities don't affect each other.

As current workaround, you have to ensure that when activating "vpn0" the untrusted connection "eth0-no-dns" on it's own doesn't provide the conflicting DNS/routes. Which would for example mean, you cannot specify the VPN gateway via DNS (because "eth0-no-dns" could not resolve it).

Anyway, this is certainly a valuable feature.

[Mincho-gaydarov (mincho-gaydarov)]

This could be fixed for example with adding checkbox "Don't use local DNS when connected to this vpn".

As far as I know the resolving via local DNS could be made before establishing VPN connection. After that when we have the IP address of the VPN server, the local DNS could be safely replaced during active vpn session.

When using multiple wireless networks, it wouldn't be convenient to disable local DNS servers for every new WiFi network. If there is option to configure the VPN in such way that no matter what connection is used it will replace local DNS, this will be easy for the user and will be more reliable than configuring every single connection in advance.

[Thomas Haller (thaller-1)]

*** Bug 765235 has been marked as a duplicate of this bug. ***

[Bgalvani (bgalvani)]

What do you guys think about this: we could add a
ipv[4,6].dns-priority property and use it to sort the IP
configurations in the DNS manager, so that the ones with a lower value
will appear first in resolv.conf. The default value would be 0 which
means "use the default value depending on connection type",
corresponding to:

  50 for VPNs
 100 for connection with default route
 200 for other connections

to keep the old behavior. The special value -1 would mean "use
exclusively this (and others with the same value)".

With this we would address both problems of (a) DNS leaks and (b)
allowing users to specify an ordering between name servers when
multiple connections are active.

[Thomas Haller (thaller-1)]

(In reply to Beniamino Galvani from comment #9)
> What do you guys think about this: we could add a
> ipv[4,6].dns-priority property and use it to sort the IP
> configurations in the DNS manager, so that the ones with a lower value
> will appear first in resolv.conf. The default value would be 0 which
> means "use the default value depending on connection type",
> corresponding to:
>
> 50 for VPNs
> 100 for connection with default route
> 200 for other connections
>
> to keep the old behavior. The special value -1 would mean "use
> exclusively this (and others with the same value)".

This sounds good. But we could allow ~all~ negative values to mean: those connections require exclusive DNS (which would disable all settings that have a value >= 0).
Then, you could also have differing priorities for exclusive connection.

Like: VPN_1a and VPN_1b could be -1, and VPN2 could be -2.

If you connect either VPN_1a or VPN_1b (but not VPN2), then only those get DNS configuration. Once you connect VPN2, also VPN_1x is disabled.

>
> With this we would address both problems of (a) DNS leaks and (b)
> allowing users to specify an ordering between name servers when
> multiple connections are active.

[Fgiudici-6 (fgiudici-6)]

I like dns-priority idea a lot.
Moreover, adding Thomas extension, I could not imagine any uncovered DNS conf user scenario.

[Thomas Haller (thaller-1)]

(In reply to Thomas Haller from comment #10)
> (In reply to Beniamino Galvani from comment #9)
> > What do you guys think about this: we could add a
> > ipv[4,6].dns-priority property and use it to sort the IP
> > configurations in the DNS manager, so that the ones with a lower value
> > will appear first in resolv.conf. The default value would be 0 which
> > means "use the default value depending on connection type",
> > corresponding to:
> >
> > 50 for VPNs
> > 100 for connection with default route

This is a bit strange. In this case, the default value depends on a quite peculiar runtime condition (i.e. whether the connection has the default route, which isn't even clear whether that means IPv4 or IPv6).
Usually our default-fallback-values are either based on some external/global (static) configuration, on some very generic (usually static) device properties, or on the (static) connection itself -- yes, there is some runtime element there too, but it's usually more clearly defined.

> > 200 for other connections

  I would choose 50 for VPN and 100 for everything else.

In face of identical priorities, policy would do what it does now: sort somehow -- and preferring the best-device.

[Bgalvani (bgalvani)]

(In reply to Thomas Haller from comment #10)
> This sounds good. But we could allow ~all~ negative values to mean: those
> connections require exclusive DNS (which would disable all settings that
> have a value >= 0).
> Then, you could also have differing priorities for exclusive connection.

(In reply to Thomas Haller from comment #12)
> I would choose 50 for VPN and 100 for everything else.
>
> In face of identical priorities, policy would do what it does now: sort
> somehow -- and preferring the best-device.

These proposals sound fine to me.

A caveat of this dns-priority thing is that when using dnsmasq the
prioritization would not work because dnsmasq forwards queries to all
name servers at the same time. There is a 'strict-order' option, but
it seems broken (according to dnsmasq's author himself) since it
basically means "if the first server time-outs, return an error to
client and then switch to the next server for future queries".

Instead, the exclusive mode would work also with dnsmasq.

There is a related request (bug 746422) to set the split mode for VPN
DNS servers according to never-default setting, which is independent
from prioritization, but probably should be handled together.

[Bgalvani (bgalvani)]

Pushed branch bg/dns-priority-bgo758772, please review.

[Thomas Haller (thaller-1)]

>> dns: use a single list to store configurations

I generally dislike the use of g_object_get_data() and GSList. Often when used, they'd better not be.

For nm_dns_plugin_update() we construct 3 separate GSList arguments (vpn_configs, dev_configs, other_configs). Which on the one hand involves additional cloning of the list.
But more importantly, the information passed there is not even sufficient, so we hack around it by passing the interface name via NM_DNS_IP_CONFIG_DATA_TAG. After that, the required information is there (for now), but we lost information which might be necessary in the future (like the sorting order).

How about replacing

  GSList *configs

with

  GArray *configs

and @configs contains a list of NMDnsIPConfigData. NMDnsIPConfigData of course gets extended to contain also the pointer to the NMIPxConfig object.

Then, drop build_plugin_config_lists(), and pass priv->configs->data directly to:

  void nm_dns_plugin_update(NMDnsPlugin *plugin,
                            const NMDnsIPConfigData *dns_config_data,
                            const NMGlobalDnsConfig *global_config,
                            const char *hostname);

[Thomas Haller (thaller-1)]

The default-value for the dns-priority cannot be configured as global default value. I think it should be.

And that that place, the fallback value for 50 or 100 should be chosen.

[Bgalvani (bgalvani)]

(In reply to Thomas Haller from comment #15)
> >> dns: use a single list to store configurations

Fixed the points above and repushed.

[Thomas Haller (thaller-1)]

>> dns: use a single array for all configurations

I like it.

_LOGE ("merge config: [ %-7s v%c %s ]",
   ^^^

>> libnm-core: add dns-priority to NMSettingIPConfig

NMSettingIPConfig:dns-priority:

+ g_param_spec_int (NM_SETTING_IP_CONFIG_DNS_PRIORITY, "", "",
+ G_MININT, G_MAXINT, 0,

How about to make the valid range int32. Yes, on most (all?) platform gint is equal to gint32, but let's be explict.
Also, because on dbus, we have it "i" (32 bit).

nm_setting_ip_config_get_dns_priority() can stay at gint (as glib already requires sizeof(int) >= 4). As you prefer...

also next commit:
priority = svGetValueInt64 (ifcfg, "IPV6_DNS_PRIORITY", 10, G_MININT32, G_MAXINT32, 0);

and here:
+ g_param_spec_int (NM_IP4_CONFIG_DNS_PRIORITY, "", "",
+ G_MININT, G_MAXINT, 0,

>> dns: use DNS priority from IP configuration

dispose() must be re-entrant:

  if (priv->configs) {
     for (i = 0; i < priv->configs->len; i++) {
     ...

     g_ptr_array_free (priv->configs, TRUE);
     priv->configs = NULL;
  }

rest lgtm

[Bgalvani (bgalvani)]

(In reply to Thomas Haller from comment #18)
> >> dns: use a single array for all configurations
> _LOGE ("merge config: [ %-7s v%c %s ]",

Fixed.

> How about to make the valid range int32. Yes, on most (all?) platform gint
> is equal to gint32, but let's be explict.
> Also, because on dbus, we have it "i" (32 bit).

Makes sense, changed.

> dispose() must be re-entrant:

Fixed this too, thanks.

Also, added a new commit "dns: don't use the global configuration to compute initial hash" and repushed.

[Thomas Haller (thaller-1)]

>> dns: use a single array for all configurations

_LOGE ("merge config: [ %-7s v%c %s ]",

still with level <error> :)

rest lgtm

[Lubomir Rintel (lkundrak)]

Looks all fine to me.

[Bgalvani (bgalvani)]

Merged to master:

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=8da3e658f7313f56928d22cfe13f9ab78cc1dd3c

[Mincho-gaydarov (mincho-gaydarov)]

According to

git tag --contains 8da3e658f7313f56928d22cfe13f9ab78cc1dd3c
and
git branch --contain 8da3e658f7313f56928d22cfe13f9ab78cc1dd3c
* master

this is still not merged.

Looking at the comments this will be included in 1.4?

[Thomas Haller (thaller-1)]

(In reply to Mincho Gaydarov from comment #23)
> According to
>
> git tag --contains 8da3e658f7313f56928d22cfe13f9ab78cc1dd3c
> and
> git branch --contain 8da3e658f7313f56928d22cfe13f9ab78cc1dd3c
> * master
>
> this is still not merged.
>
> Looking at the comments this will be included in 1.4?

it is merged to "master", as you see in `git branch --contains`.
There is no tag, because there was no upstream release since then. 1.4.0 will be the next upstream release which contains this feature.

[Bgalvani (bgalvani)]

*** Bug 738647 has been marked as a duplicate of this bug. ***

[Sebastien Bacher (seb128)]

The upstream bug got closed as resolved in 2016