network-manager[-openvpn] doesn't handle properly routes pushed by OpenVPN 2.1_Rc7

[Ulf Andersson]

Openvpn 2.1_rc7 was included in the Ubuntu 8.04 hardy 2008-02-22 and
after that upgrade openvpn does not route to tap or tun interfaces.
Sometimes I get the "Route Error" - but not always. I'm using Ubuntu 8.04 AMD64 Alteranative

Update:
From the latest comments it appears that the routes are pushed correctly by OpenVPN, but that network-manager or network-manager-openvpn undo the route changes.

[Andreas (andreas-kotowicz)]

same here with OpenVPN 2.1_rc7-1ubuntu3

ovpn-openvpn[8583]: TUN/TAP device tap0 opened
ovpn-openvpn[8583]: TUN/TAP TX queue length set to 100
ovpn-openvpn[8583]: ifconfig tap0 172.16.2.141 netmask 255.255.254.0 mtu 1500 broadcast 172.16.3.255
avahi-daemon[6859]: Joining mDNS multicast group on interface tap0.IPv4 with address 172.16.2.141.
avahi-daemon[6859]: New relevant interface tap0.IPv4 for mDNS.
avahi-daemon[6859]: Registering new address record for 172.16.2.141 on tap0.IPv4.
avahi-daemon[6859]: Withdrawing address record for 172.16.2.141 on tap0.
avahi-daemon[6859]: Leaving mDNS multicast group on interface tap0.IPv4 with address 172.16.2.141.
avahi-daemon[6859]: Interface tap0.IPv4 no longer relevant for mDNS.
avahi-daemon[6859]: Joining mDNS multicast group on interface tap0.IPv4 with address 172.16.2.141.
avahi-daemon[6859]: New relevant interface tap0.IPv4 for mDNS.
avahi-daemon[6859]: Registering new address record for 172.16.2.141 on tap0.IPv4.
ovpn-openvpn[8583]: PLUGIN_CALL: POST plugins/openvpn-down-root.so/PLUGIN_UP status=0
ovpn-openvpn[8583]: scripts/client.up tap0 1500 1590 172.16.2.141 255.255.254.0 init
NetworkManager: <info> tap0: Device is fully-supported using driver '(null)'.
NetworkManager: <info> nm_device_init(): waiting for device's worker thread to start
NetworkManager: <info> nm_device_init(): device's worker thread started, continuing.
NetworkManager: <info> Now managing wired Ethernet (802.3) device 'tap0'.
NetworkManager: <info> Deactivating device tap0.
avahi-daemon[6859]: Withdrawing address record for 172.16.2.141 on tap0.
avahi-daemon[6859]: Leaving mDNS multicast group on interface tap0.IPv4 with address 172.16.2.141.
avahi-daemon[6859]: Interface tap0.IPv4 no longer relevant for mDNS.
NetworkManager: <info> Will activate wired connection 'tap0' because it now has a link.
NetworkManager: <info> Will activate connection 'tap0'.
NetworkManager: <info> Device tap0 activation scheduled...
NetworkManager: <info> Deactivating device eth1.
dhclient: There is already a pid file /var/run/dhclient.eth1.pid with pid 8478
dhclient: killed old client process, removed PID file
ovpn-openvpn[8583]: route add -net 130.60.230.177 netmask 255.255.255.255 gw 192.168.1.1
ovpn-openvpn[8583]: route add -net 0.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[8583]: ERROR: Linux route add command failed: shell command exited with error status: 7
ovpn-openvpn[8583]: route add -net 128.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[8583]: ERROR: Linux route add command failed: shell command exited with error status: 7
ovpn-openvpn[8583]: chroot to '/etc/openvpn' and cd to '/' succeeded
ovpn-openvpn[8583]: GID set to daemon
ovpn-openvpn[8583]: UID set to nobody
ovpn-openvpn[8583]: Initialization Sequence Completed

[Andreas (andreas-kotowicz)]

I just tried using all other openvpn versions: 2.1~rc7-1ubuntu2, 2.1~rc7-1ubuntu1, 2.1~rc4-2 and 2.0.9-8

I always keep getting the same error:

avahi-daemon[6859]: Withdrawing address record for 192.168.1.3 on eth1.
avahi-daemon[6859]: Leaving mDNS multicast group on interface eth1.IPv4 with address 192.168.1.3.
avahi-daemon[6859]: Interface eth1.IPv4 no longer relevant for mDNS.
ovpn-openvpn[10940]: route add -net 130.60.230.177 netmask 255.255.255.255 gw 192.168.1.1
ovpn-openvpn[10940]: ERROR: Linux route add command failed: shell command exited with error status: 7
ovpn-openvpn[10940]: route add -net 0.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[10940]: ERROR: Linux route add command failed: shell command exited with error status: 7
ovpn-openvpn[10940]: route add -net 128.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[10940]: ERROR: Linux route add command failed: shell command exited with error status: 7
ovpn-openvpn[10940]: chroot to '/etc/openvpn' and cd to '/' succeeded
ovpn-openvpn[10940]: GID set to daemon
ovpn-openvpn[10940]: UID set to nobody
ovpn-openvpn[10940]: Initialization Sequence Completed
ovpn-openvpn[10940]: write UDPv4 []: Network is unreachable (code=101)
last message repeated 3 times
Saturn avahi-daemon[6859]: Withdrawing address record for fe80::20e:35ff:fe4c:63d7 on eth1.
Saturn avahi-daemon[6859]: Leaving mDNS multicast group on interface eth1.IPv6 with address fe80::20e:35ff:fe4c:63d7.
Saturn avahi-daemon[6859]: Interface eth1.IPv6 no longer relevant for mDNS.
Saturn NetworkManager: <info> Activation (tap0) started...
Saturn NetworkManager: <info> Activation (tap0) Stage 1 of 5 (Device Prepare) scheduled...
Saturn NetworkManager: <info> Activation (tap0) Stage 1 of 5 (Device Prepare) started...
Saturn NetworkManager: <info> Activation (tap0) Stage 2 of 5 (Device Configure) scheduled...
Saturn NetworkManager: <info> Activation (tap0) Stage 1 of 5 (Device Prepare) complete.
Saturn NetworkManager: <info> Activation (tap0) Stage 2 of 5 (Device Configure) starting...
Saturn NetworkManager: <info> Activation (tap0) Stage 2 of 5 (Device Configure) successful.
Saturn NetworkManager: <info> Activation (tap0) Stage 3 of 5 (IP Configure Start) scheduled.
Saturn NetworkManager: <info> Activation (tap0) Stage 2 of 5 (Device Configure) complete.
Saturn NetworkManager: <info> Activation (tap0) Stage 3 of 5 (IP Configure Start) started...
Saturn NetworkManager: <info> Old device 'tap0' activating, won't change.
Saturn NetworkManager: <info> Activation (tap0) Beginning DHCP transaction.
Saturn dhclient: There is already a pid file /var/run/dhclient.tap0.pid with pid 134519072
Saturn NetworkManager: <info> Activation (tap0) Stage 3 of 5 (IP Configure Start) complete.
Saturn NetworkManager: <info> DHCP daemon state is now 12 (successfully started) for interface tap0
Saturn NetworkManager: <info> DHCP daemon state is now 1 (starting) for interface tap0
Saturn dhclient: DHCPDISCOVER on tap0 to 255.255.255.255 port 67 interval 7
Saturn ovpn-openvpn[10940]: write UDPv4 []: Network is unreachable (code=101)
Saturn NetworkManager: <info> Old device 'tap0' activating, won't change.

BUT: ope...

[Andreas (andreas-kotowicz)]

This is definitely NOT an openvpn problem but an openvpn & NetworkManager problem.
If I kill NetworkManager and NetworkManagerDispatcher, openvpn runs without any problems:

ovpn-openvpn[11797]: TUN/TAP device tap0 opened
ovpn-openvpn[11797]: TUN/TAP TX queue length set to 100
ovpn-openvpn[11797]: ifconfig tap0 172.16.2.142 netmask 255.255.254.0 mtu 1500 broadcast 172.16.3.255
ovpn-openvpn[11797]: PLUGIN_CALL: POST plugins/openvpn-down-root.so/PLUGIN_UP status=0
ovpn-openvpn[11797]: scripts/client.up tap0 1500 1590 172.16.2.142 255.255.254.0 init
avahi-daemon[6859]: Joining mDNS multicast group on interface tap0.IPv4 with address 172.16.2.142.
avahi-daemon[6859]: New relevant interface tap0.IPv4 for mDNS.
avahi-daemon[6859]: Registering new address record for 172.16.2.142 on tap0.IPv4.
avahi-daemon[6859]: Withdrawing address record for 172.16.2.142 on tap0.
avahi-daemon[6859]: Leaving mDNS multicast group on interface tap0.IPv4 with address 172.16.2.142.
avahi-daemon[6859]: Interface tap0.IPv4 no longer relevant for mDNS.
avahi-daemon[6859]: Joining mDNS multicast group on interface tap0.IPv4 with address 172.16.2.142.
avahi-daemon[6859]: New relevant interface tap0.IPv4 for mDNS.
avahi-daemon[6859]: Registering new address record for 172.16.2.142 on tap0.IPv4.
ovpn-openvpn[11797]: route add -net 130.60.230.177 netmask 255.255.255.255 gw 192.168.1.1
ovpn-openvpn[11797]: route add -net 0.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[11797]: route add -net 128.0.0.0 netmask 128.0.0.0 gw 172.16.3.254
ovpn-openvpn[11797]: chroot to '/etc/openvpn' and cd to '/' succeeded
ovpn-openvpn[11797]: GID set to daemon
ovpn-openvpn[11797]: UID set to nobody
ovpn-openvpn[11797]: Initialization Sequence Completed

[Ulf Andersson (uanderss)]

I also tested to only kill NetworkManager and openvpn is running correctly.
When I ran NetworkManger-Openvpn it works but the description of the wired network in the tray-icon
got a really long name and then it did not work.

So, I agree, this is a openvpn and networkmanager problem.

[Alessandro Gervaso (gervystar)]

I confirm this because nor the plugin nor the daemon are able to set the proper routes up.

[Alessandro Gervaso (gervystar)]

Seems to be working again since two weeks ago.

[David Miller (david3d)]

I'm still seeing bad routes being setup by NetworkManager-Openvpn under an up to date 8.04 install. If I drop down to a command line and connect to the vpn using sudo openvpn --config <config file> everything works fine. But when I connect to the vpn via NetworkManager the only computer that I can get to over the vpn is the system that the vpn is running on....I can't get anything back from the remote LAN subnet. Tonight I'll try to attach the output of routes from when I connect via the network manager and when I connect via the command line.

[David Miller (david3d)]

Sorry for the delay here's my routing tables but first a little background info. The vpn is supposed to push two routes to the client. A route to the companies mail server and a route to the local LAN.

Connected via the command line.....vpn connection works correctly.

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
mail.domain.com 172.20.1.129 255.255.255.255 UGH 0 0 0 tap0
172.20.1.128 * 255.255.255.128 U 0 0 0 tap0
172.20.0.0 172.20.1.129 255.255.255.0 UG 0 0 0 tap0
192.168.0.0 * 255.255.255.0 U 0 0 0 wlan0
link-local * 255.255.0.0 U 1000 0 0 wlan0
default DD-Internet.hom 0.0.0.0 UG 0 0 0 wlan0

Connected via network-manager-openvpn plugin. Here you can see we are not getting any routes from the vpn server and the tap interface is now the default gateway.

Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
vpn.domain.com 192.168.0.1 255.255.255.255 UGH 0 0 0 wlan0
192.168.0.0 * 255.255.255.0 U 0 0 0 wlan0
link-local * 255.255.0.0 U 1000 0 0 wlan0
default * 0.0.0.0 U 0 0 0 tap0

[Alessandro Gervaso (gervystar)]

My previous reply was not clear, sorry: I just meant that it was working again from the command line.
Using the network-manager plugin I get the same results as David Miller.

[Thomas Jacob (jacob-internet24)]

I'd like to second Alessandro and David, with a tap setup, network-manager-openvpn is not able to setup all the push routes that
the OpenVPN server is delivering and this makes this plugin pretty much unusable.

The problem doesn't seem to be that that routes are not set up per se, but that someone (else? presumably network-manager or network-manager-openvpn) deletes them afterwards, but strangely only some of them (looks like only the routes that use the tap-device). You can nicely watch this happening using "ip monitor".

In general, network-manager(-openvpn) seems to be broken by design since its inception. How hard can it be to build a GUI that is able
to do what sudo /etc/init.d/openvpn <CONFIG> start does on the command line? Why invent a complicated mechanism that
doesn't do anything that openvpn doesn't do better itself? Why not simply use OpenVPN config files, maybe getting a password from the user
and then let openvpn do its job. But sure, it's easy to complain, when you don't actually contribute.

[Alexander Sack (asac)]

could someone please test whether this issue still exists in intrepid?

[Alexander Sack (asac)]

unless i see evidence that its core-NM issue, I assume its a vpn plugin problem. Testing with NM 0.7 would help to know more.

[Thierry Carrez (ttx)]

Removing openvpn from the scope, following comments that openvpn in itself works properly.
Please re-add it if analysis shows that the problem does indeed come from openvpn.

[Dimitrios Symeonidis (azimout)]

guys, is this still an issue for any of you?

[Diego Rivera (diego-rivera-rbxglobal)]

Yes, this is still an issue. I can confirm what our friend said earlier about routes being correctly added by OpenVPN and subsequently being removed (presumably by NetworkManager or the NM-OpenVPN plugin).

In my view the problem is that the options do not permit the selection of the correct behavior. There are 3 possible scenarios:

1) User explicitly chooses to use the server as the default gateway ("user-specified-routing" only makes sense in the context of route-exclusions)
2) User does NOT explicitly choose to use the server as the default gateway, and wants to specify which targets to route manually over the tunnel (current behavior)
3) Same as case 2, but add to that the acceptance of server-provided routing info (this is the case not functioning). In this case, manual routes may also make sense as manual, forced exclusions or routes augmenting the routes received from the server (i.e. 10.0.0.0/8, !10.5.0.0/24).

I'm willing to work on fixing this, but I need insights as to where in NM or NM-OpenVPN the routes are removed so I can add the logic to selectively do that only when appropriate.

Anyone care to help me get started?

[Diego Rivera (diego-rivera-rbxglobal)]

Ok I've located the culprit logic. It's in NetworkManagerSystem.c - the logic first removes all routes attached to the new VPN interface, and then proceeds to force-feed what it believes to be the correct routing configurations. I think what's appropriate here is the addition of a flag in the DBUS messages which indicates the behavior to be followed. If the flag is missing, the default (i.e. current) behavior is followed - otherwise we can respect routing as implemented by the VPN backend (desired), or some combination of the two.

I'll get cracking on this and see if I can come up with a small, simple patch for the problem.

Please note: for now, I'm working only on 8.04 (NM 0.6.6). After that, I'll have a look at newer versions and see if the patch would need work.

Cheers.

[Diego Rivera (diego-rivera-rbxglobal)]

Here's my first crack at a patch. It's for Ubuntu 8.04LTS (package versions are referenced in each patch filename).

It has the GUI change for the OpenVPN stuff only. For PPTP the new "delegating routing mode" doesn't really make sense since PPTP doesn't have a mechanism (that I know of) to transfer routing information to a client in a portable way. Thus - it's either default route, or manual routing (i.e. the old, "bad" behavior).

For VPNC the story is different - IPSec mode_cfg does permit the communication of routing information to the client. I'm not sure if vpnc supports this, but this is the other part of the GUI that might need some work.

The OpenVPN stuff works as advertised. I didn't check the file import/export - I'm too tired today :)

The change consists in adding a configuration option called "route_mode" which has (currently) two values (possibly more in the future): 0 or absent = current, "broken" behavior, 1 = new behavior (delegate routing to the VPN client). The GUI portion to manipulate this configuration setting is completed only for OpenVPN.

[Diego Rivera (diego-rivera-rbxglobal)]

Ok... here's my second (actually, third! :D), cleaner crack at the same patch. The above patch had a couple of bugs in deleting VPN connections (wouldn't be properly deleted), and wouldn't import the new route_mode value from files.

Again - it still only covers network-manager and network-manager-openvpn. The other network-manager vpn modules (vpnc and pptp) are also modified, but only to maintain binary compatibility. In particular, this new functionality makes sense *at most* for vpnc, since PPTP doesn't have a portable mechanism to provide routing information like OpenVPN does (for vpnc/IPSec, mod_cfg is able to provide said routing info).

Cheers.

[Tony Espy (awe)]

Diego, thanks for the attached patches. asac is on vacation, so I'll attempt to field this one for now...

In order for your patches to land in Hardy 8.04, they'd have to be released via the SRU process:

https://wiki.ubuntu.com/StableReleaseUpdates

I'm not saying it's impossible to land your changes as an SRU, but as your patches contain changes to both NetworkManager and the applet, they may be seen as risky.

Also the patches as attached would need to be re-factored to separate out changes to the files in the debian directory, and to utilize the correct patch system ( eg. quilt for NetworkManager ).

Ideally, the best place for your changes to land would be the current release under development ( Karmic ), which unfortunately is about to hit FeatureFreeze ( tomorrow ).

Since what you've really done is add a new feature which adds missing functionality, another possibility would be to try and work directly with upstream to get your patches integrated into NM 0.8 which is currently under development. If done in a timely manner, there's potential that your changes could land in Karmic, as we may continue to pull upstream snapshots of NM 0.8 and the VPN plugins as we try and finalize the release.

You can contact the upstream developers, and myself and asac on #nm on the Ubuntu IRC server.

Here are the URLs for the upstream git trees for NM, the applet, and the vpn plugins:

git://anongit.freedesktop.org/NetworkManager/NetworkManager
git://git.gnome.org/network-manager-applet
git://git.gnome.org/network-manager-openvpn
git://git.gnome.org/network-manager-pptp
git://git.gnome.org/network-manager-vpnc

asac may have a different opinion, so you might want to try and touch base with him next week when he returns from vacation, but I think the upstream approach may be the best way to go.

[Diego Rivera (diego-rivera-rbxglobal)]

I sort of expected this to be the case - I'll read up on how to refactor those patches as you metion.

However, the important part is that it the patches are built and designed such that if you install the patched version over existing configs, nothing changes functionally - i.e. you have to actually want to use the new functionality for it to be activated. If not, then everything hums along the same way it did before the patch. This is intentionally so since I understand what LTS is all about - it's about configuration management and stability.

I'll try to get it factored for today, but you're right - it's unlikely that I'll be able to. Should I be able to, what's the most expedited way to get the ball rolling to attempt to fit this in prior to feature freeze?

Thanks for the info!

[Diego Rivera (diego-rivera-rbxglobal)]

This defect severely limits the functionality of the OpenVPN client. PPTP is unaffected since the functionality is consistent with the available feature set. VPNC might be a case similar to OpenVPN.

[Tony Espy (awe)]

Diego, as I pointed out above, the FF for Karmic is tomorrow. Exceptions are possible, however at this stage of the game, the approach I mentioned above ( ie. working with upstream ), is probably your best bet.

If you're able to re-factor your patches against the upstream git trees, and create git patches, then they can just be emailed to the NetworkManager email list ( see http://mail.gnome.org/mailman/listinfo/networkmanager-list ). That should get the conversation started with upstream.

Again, as for landing your patches in Hardy, I'd wait and get asac's opinion before re-factoring your patches against NM 0.6.6. I understand your point about nothing changing unless you explicitly enable the functionality, which is the right approach, however I still have doubts that your changes would meet SRU approval.

[Thomas Hood (jdthood)]

@Ulf, Tony, Diego: Is there any improvement in Ubuntu 12.04?

[Launchpad Janitor (janitor)]

[Expired for network-manager (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for network-manager-openvpn (Ubuntu) because there has been no activity for 60 days.]