OpenVPN causes reboot failure on Xenial in AWS

[Scott Crooks]

The problem actually has to do with the way `systemd` is configured in the latest version of Ubuntu 16.04. Setting up a typical OpenVPN configuration, and rebooting causes a Instance Status check failure in AWS EC2. The reason is because the file `/lib/systemd/system/openvpn@.service` is missing the following line:

After=network.target network-online.target

The entire [Unit] section then looks like:

[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service
Before=systemd-user-sessions.service
After=network.target network-online.target
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

This ensures that routes needed for OpenVPN configurations get called after the main networking stack.

See issues here:

- https://forums.aws.amazon.com/thread.jspa?messageID=719570
- https://sourceforge.net/p/openvpn/mailman/message/35078170/

[Simon Déziel (sdeziel)]

Scott, could you please provide some information about the routing/IP configuration before the reboot?

[Scott Crooks (scott-crooks)]

Simon,

Sure. Would you prefer a Github Gist, or can I paste the outputs here directly?

[Scott Crooks (scott-crooks)]

Simon,

Also, which commands do you want to see? Just `ip route` ?

[Simon Déziel (sdeziel)]

If you could attach "ip addr" and "ip route" output here that would be good, thanks.

[Scott Crooks (scott-crooks)]

Simon,

The setup I have is running two OpenVPN daemons on the same port (1618) but on different protocols (TCP/UDP). The post that Chris made in the AWS forum above had a much simpler, single virtual interface setup.

root@ip-10-0-0-9:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 06:cc:43:0f:66:e9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.9/28 brd 10.0.0.15 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::4cc:43ff:fe0f:66e9/64 scope link
       valid_lft forever preferred_lft forever
3: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.255.248.1/23 brd 10.255.249.255 scope global tun0
       valid_lft forever preferred_lft forever
4: tun1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.255.250.1/23 brd 10.255.251.255 scope global tun1
       valid_lft forever preferred_lft forever

root@ip-10-0-0-9:~# ip route
default via 10.0.0.1 dev eth0
10.0.0.0/28 dev eth0 proto kernel scope link src 10.0.0.9
10.255.248.0/23 dev tun0 proto kernel scope link src 10.255.248.1
10.255.250.0/23 dev tun1 proto kernel scope link src 10.255.250.1

root@ip-10-0-0-9:~# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:1618 0.0.0.0:* LISTEN 2875/openvpn
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1059/sshd
tcp6 0 0 :::22 :::* LISTEN 1059/sshd
udp 0 0 0.0.0.0:68 0.0.0.0:* 876/dhclient
udp 0 0 0.0.0.0:1618 0.0.0.0:* 2874/openvpn
Active UNIX domain sockets (only servers)
Proto RefCnt Flags Type State I-Node PID/Program name Path
unix 2 [ ACC ] STREAM LISTENING 8782 1/init /run/lvm/lvmpolld.socket
unix 2 [ ACC ] STREAM LISTENING 20818 2974/systemd /run/user/1000/systemd/private
unix 2 [ ACC ] SEQPACKET LISTENING 8864 1/init /run/udev/control
unix 2 [ ACC ] STREAM LISTENING 8783 1/init /run/systemd/journal/stdout
unix 2 [ ACC ] STREAM LISTENING 9033 1/init /run/lvm/lvmetad.socket
unix 2 [ ACC ] STREAM LISTENING 10909 1/init /run/acpid.socket
unix 2 [ ACC ] STREAM LISTENING 20850 3012/0 /tmp/ssh-zhPNN6x7d9/agent.3012
unix 2 [ ACC ] STREAM LISTENING 10899 1/init ...

[Simon Déziel (sdeziel)]

Thanks Scott. Could you also attach the OpenVPN configs since they apparently don't match the gist [1].

Thanks

1: https://gist.github.com/sc250024/001e05d64cd02fe746ae1772baccd24a

[Scott Crooks (scott-crooks)]

Simon,

The configuration was more or less the same, just different port numbers. Here they are for documentation purposes. Again, the only differences are the protocol, and the IP pools.

UDP Config:
port 1618
proto udp
dev tun
ca /etc/openvpn/openvpnudp-files/ca.crt
cert /etc/openvpn/openvpnudp-files/server.crt
key /etc/openvpn/openvpnudp-files/server.key
dh /etc/openvpn/openvpnudp-files/dh3072.pem
push "redirect-gateway def1 bypass-dhcp"
server 10.255.248.0 255.255.254.0
remote-cert-eku "TLS Web Client Authentication"
ifconfig-pool-persist /etc/openvpn/openvpnudp-files/ipp.txt
keepalive 10 120
comp-lzo yes
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 5
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"
plugin /usr/lib/openvpn/openvpn-auth-ldap.so "/etc/openvpn/openvpnudp-files/auth-ldap.conf"
username-as-common-name
cipher AES-256-CBC
topology subnet
log-append /var/log/openvpn.log
tls-auth /etc/openvpn/openvpnudp-files/tls.key 0
key-direction 0
tls-version-min 1.2
auth SHA512
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
crl-verify /etc/openvpn/openvpnudp-files/crl.pem

TCP Config:
port 1618
proto tcp
dev tun
ca /etc/openvpn/openvpntcp-files/ca.crt
cert /etc/openvpn/openvpntcp-files/server.crt
key /etc/openvpn/openvpntcp-files/server.key
dh /etc/openvpn/openvpntcp-files/dh3072.pem
push "redirect-gateway def1 bypass-dhcp"
server 10.255.250.0 255.255.254.0
remote-cert-eku "TLS Web Client Authentication"
ifconfig-pool-persist /etc/openvpn/openvpntcp-files/ipp.txt
keepalive 10 120
comp-lzo yes
persist-key
persist-tun
status /var/log/openvpn-status.log
verb 5
push "dhcp-option DNS 208.67.222.222"
push "dhcp-option DNS 208.67.220.220"
plugin /usr/lib/openvpn/openvpn-auth-ldap.so "/etc/openvpn/openvpntcp-files/auth-ldap.conf"
username-as-common-name
cipher AES-256-CBC
topology subnet
log-append /var/log/openvpn.log
tls-auth /etc/openvpn/openvpntcp-files/tls.key 0
key-direction 0
tls-version-min 1.2
auth SHA512
tls-cipher TLS-ECDHE-RSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-ECDSA-WITH-AES-128-GCM-SHA256:TLS-ECDHE-RSA-WITH-AES-256-GCM-SHA384:TLS-DHE-RSA-WITH-AES-256-CBC-SHA256
crl-verify /etc/openvpn/openvpntcp-files/crl.pem

[Scott Crooks (scott-crooks)]

Greetings,

Any status on this? The thread I opened on the openvpn-user mailing list has confirmed as well that this is an issue. See here: http://article.gmane.org/gmane.network.openvpn.user/36909

[Simon Déziel (sdeziel)]

Hi Scott,

I just finished migrating my VPN to Xenial. I cannot reproduce your issue using a static IP configuration in /etc/network/interfaces. I'll try to set things up with DHCP and see how it goes.

[Scott Crooks (scott-crooks)]

Simon,

Yes, the instance I used as a test was using DHCP. AWS is a bit odd in that sense since, for all intents and purposes, your instance has a static internal IP address that persists across reboots. However, the instance thinks it's using DHCP. It would be as if you configured DHCP on the OpenVPN server, but configured the LAN's DHCP server to assign only one address to your OpenVPN server.

[WoJ (g-w)]

I confirm that the issue is present with the AMI Ubuntu image 20160516.1. I systematically reproduce it by

 - launching an instance of the AMI image on EC2
 - apt update ; apt install openvpn
 - configuring the OpenVN server with any configuration (static key for instance https://openvpn.net/index.php/open-source/documentation/miscellaneous/78-static-key-mini-howto.html)
 - enable OpenVPN at boot
 - reboot

The instance fails to configure its network for the reason described by the OP.

[Samuli Seppänen (samuli-seppanen)]

This link might be of interest:

<https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/>

Much of the same info can be viewed with

$ man 7 systemd.special

It seems that network-online.target should be used instead of network.target. I think fixing this in openvpn.service instead of openvpn@.service should work, but I have not tried it, nor have I actually tried reproducing this bug myself.

[Scott Harney (scotth-5)]

I tested that theory of modifying openvpn.service with network-online.target to no avail. Can confirm this issue in ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20160610 (ami-d06a90b0) as well.

Will try modifying openvpn.@service as suggested by OP to see if that resolves.

[Scott Harney (scotth-5)]

Doing as the OP suggested does work. the following modification of lib/systemd/system/openvpn@.service allows the AWS xenial instance to reboot with openvpn service enabled

$ diff -u /lib/systemd/system/openvpn@.service roles/openvpn_server/files/openvpn@.service
--- /lib/systemd/system/openvpn@.service 2016-02-02 06:33:30.000000000 -0600
+++ roles/openvpn_server/files/openvpn@.service 2016-06-27 15:55:19.839586907 -0500
@@ -3,6 +3,7 @@
 PartOf=openvpn.service
 ReloadPropagatedFrom=openvpn.service
 Before=systemd-user-sessions.service
+After=network.target network-online.target
 Documentation=man:openvpn(8)
 Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
 Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Samuli Seppänen (samuli-seppanen)]

Upstream OpenVPN has two unit files, one for servers, one for clients:

<https://community.openvpn.net/openvpn/ticket/462>
<https://github.com/OpenVPN/openvpn/tree/master/distro/systemd>

Have you tested whether these unit files work better?

[Simon Déziel (sdeziel)]

There is a nice explanation of the bug in the duplicate LP: #1598522.

[Malte Zacharias (mz-basis+ubuntu)]

I can confirm the above reports as well as the description in #1595822.
OpenVPN is started too early in the boot process. Cloud-Init will default to configure tun0 instead of the guests main network interface via dhcp. This breaks the cloud-init process.

[Stas (ivaschenko-stas)]

I have just tried <https://github.com/OpenVPN/openvpn/tree/master/distro/systemd>
It works OK, however, they assume that server configuration is in /etc/openvpn/server/<name>.conf, in current ubuntu package it is: /etc/openvpn/<name>.conf that is the only issue. Everything else is great. So if you take those files from upstream, ppl will have to change directory structure a bit, not sure if that is a big issue or not.

[Scott Crooks (scott-crooks)]

Is there a timeline for this fix? It seems like a pretty simple fix, and it's a bit nerve-racking every time I have to reboot an OpenVPN server :)

[David Sommerseth (dazo)]

I am the one who added the systemd unit files to upstream OpenVPN. These files are now getting a few updates [1] and will ship as the recommended unit files in the coming OpenVPN 2.4 release.

The OpenVPN team would appreciate if Ubuntu was shipping the upstream version of these unit files. We try to make all systemd based distributions use the same ones, as that gives less maintenance burden on package maintainers as well as a consistent behaviour across all distributions. Which again helps the support guys on #openvpn @ FreeNode when something isn't working.

With that said, I am happy to see that our versions resolves this issue. If anyone is interested in helping out testing, fixing and improving these unit files further, please do not hesitate getting in touch with us on #openvpn-devel @ FreeNode.

[1] <https://<email address hidden>/msg12764.html>