[SRU] OpenVPN will not reload due to misconfigured .service file

Bug #1868127 reported by John Runyon on 2020-03-19
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
openvpn (Ubuntu)
Medium
Lucas Kanashiro
Bionic
Medium
Lucas Kanashiro
Eoan
Medium
Lucas Kanashiro
Focal
Medium
Lucas Kanashiro

Bug Description

[Impact]

The command 'systemctl reload openvpn @ $ foo' is broken for a while and no one have reported that. Users should not be using it a lot. After some investigation we notice the restart command does basically the same thing, and users should be using restart and not reload. Our proposal here is to drop the reload support (it is not mandatory) to avoid users getting errors while trying to use it.

[Test Case]

* Setup an OpenVPN server
* Try to reload the service: $ systemctl reload openvpn@$foo

$ sudo systemctl reload openvpn@server
Job for <email address hidden> failed.
See "systemctl status <email address hidden>" and "journalctl -xe" for details.

[Regression Potential]

The legacy systemd unit file were changed (openvpn.service and openvpn@.service), so if a regression is going to happen is there. We did not change any existent config, we simply removed the reload related config.

[Original Description]

OpenVPN will not reload due to misconfigured .service file

You remove CAP_KILL (by not listing it in CapabilityBoundingSet). OpenVPN should be configured to drop privileges, which means that it will no longer be running as root, while kill is running with root, which means CAP_KILL is required to send a signal. It either needs to be listed in CapabilityBoundingSet, or (preferably) "ExecReload=/bin/kill -HUP $MAINPID" needs to become "ExecReload=+/bin/kill -HUP $MAINPID"

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: openvpn 2.4.4-2ubuntu1.3
ProcVersionSignature: Ubuntu 4.15.0-91.92-generic 4.15.18
Uname: Linux 4.15.0-91-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.12
Architecture: amd64
Date: Thu Mar 19 10:48:18 2020
InstallationDate: Installed on 2018-05-02 (686 days ago)
InstallationMedia: Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: openvpn
UpgradeStatus: No upgrade log present (probably fresh install)
modified.conffile..etc.openvpn.update-resolv-conf: [deleted]

Related branches

John Runyon (failureyousuck) wrote :
John Runyon (failureyousuck) wrote :

(PS: issue also exists in 2.4.7-1ubuntu2)

Simon Déziel (sdeziel) wrote :

The "ExecReload=+/bin/kill" way of reloading without needing extra caps seems sensible. That said, I'm wondering what's the use case for a reload instead of a restart as man openvpn(8) describes what happens on SIGHUP:

SIGNALS
  SIGHUP Cause OpenVPN to close all TUN/TAP and network connections, restart, re-read the
  configuration file (if any), and reopen TUN/TAP and network connections.

I'm wondering how it is possible when OpenVPN downgraded UID/GID? Maybe it works when using the openvpn-plugin-down-root.so plugin?

tags: added: server-next
Changed in openvpn (Ubuntu):
status: New → Triaged
John Runyon (failureyousuck) wrote :

That is a very good point. I never noticed it but it looks like my OpenVPN is actually failing to reload, exiting, and being restarted by systemd. It looks like I'll need to modify my 'overlay' service to use SIGUSR1 instead.

Changed in openvpn (Ubuntu Bionic):
status: New → Triaged
Changed in openvpn (Ubuntu Eoan):
status: New → Triaged
Changed in openvpn (Ubuntu Focal):
status: New → Triaged

Executing the ExecReload= command with full privileges (adding '+') indeed fixes the reload failure (I ran my tests in a Bionic and Focal VMs but it should apply to the other releases):

May 26 10:01:41 openvpn-reload kill[1764]: kill: (1738): Operation not permitted

I mean the command 'systemctl reload openvpn@<server>' does not fail (returns 0). However, after checking the journal log I found the following error messages:

May 26 10:30:57 openvpn-reload ovpn-server[10626]: SIGHUP[hard,] received, process restarting
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --dh fails with 'dh.pem': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --ca fails with 'ca.crt': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --cert fails with 'server.crt': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --key fails with 'server.key': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --tls-auth fails with 'ta.key': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --writepid fails with '/run/openvpn/server.pid': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --status fails with '/var/log/openvpn/openvpn-status.log': Permission denied (errno=13)
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: Please correct these errors.
May 26 10:30:57 openvpn-reload ovpn-server[10626]: Use --help for more information.

As discussed above I am not quite sure about the practical difference between restarting the service and sending the SIGHUP signal to the process. The upstream message when handling this signal is "process restarting". Since defining a reload command is optional and the difference between restart and reload is not clear, we may want to remove the ExecReload= from this unit file. Maybe run the ExecReload= with full privileges ('+') in the supported releases to not remove this feature and avoid the failure, and remove the ExecReload= from the development release (stop supporting reload)?

Download full text (4.5 KiB)

I'm not sure there's even any point to adding the '+': an error will occur
either way, whether it's in kill or in OpenVPN. At least when it appears
from kill it's obvious something went wrong (even if it's not obvious
what...).

Personally I think removing reload is the right call going forward - having
perused the man page more thoroughly, neither SIGHUP nor SIGUSR1 accomplish
what I would call a 'reload'. But that's just my opinion.

On Tue, May 26, 2020, 14:00 Lucas Kanashiro <email address hidden>
wrote:

> Executing the ExecReload= command with full privileges (adding '+')
> indeed fixes the reload failure (I ran my tests in a Bionic and Focal
> VMs but it should apply to the other releases):
>
> May 26 10:01:41 openvpn-reload kill[1764]: kill: (1738): Operation not
> permitted
>
> I mean the command 'systemctl reload openvpn@<server>' does not fail
> (returns 0). However, after checking the journal log I found the
> following error messages:
>
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: SIGHUP[hard,] received,
> process restarting
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --dh
> fails with 'dh.pem': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --ca
> fails with 'ca.crt': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --cert
> fails with 'server.crt': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --key
> fails with 'server.key': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error:
> --tls-auth fails with 'ta.key': Permission denied (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error:
> --writepid fails with '/run/openvpn/server.pid': Permission denied
> (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: --status
> fails with '/var/log/openvpn/openvpn-status.log': Permission denied
> (errno=13)
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Options error: Please
> correct these errors.
> May 26 10:30:57 openvpn-reload ovpn-server[10626]: Use --help for more
> information.
>
> As discussed above I am not quite sure about the practical difference
> between restarting the service and sending the SIGHUP signal to the
> process. The upstream message when handling this signal is "process
> restarting". Since defining a reload command is optional and the
> difference between restart and reload is not clear, we may want to
> remove the ExecReload= from this unit file. Maybe run the ExecReload=
> with full privileges ('+') in the supported releases to not remove this
> feature and avoid the failure, and remove the ExecReload= from the
> development release (stop supporting reload)?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1868127
>
> Title:
> OpenVPN will not reload due to misconfigured .service file
>
> Status in openvpn package in Ubuntu:
> Triaged
> Status in openvpn source package in Bionic:
> Triaged
> Status in openvpn source package in Eoan:
> Triaged
> Sta...

Read more...

I proposed to add the '+' in the supported releases to avoid a failure while users try to reload the service:

$ sudo systemctl reload openvpn@server
Job for <email address hidden> failed because the control process exited with error code.
See "systemctl status <email address hidden>" and "journalctl -xe" for details.
$ echo $?
1
$ sudo systemctl status openvpn@server
● <email address hidden> - OpenVPN connection to server
   Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
   Active: active (running) (Result: exit-code) since Tue 2020-05-26 17:11:33 -03; 24s ago
     Docs: man:openvpn(8)
           https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
           https://community.openvpn.net/openvpn/wiki/HOWTO
  Process: 9548 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=1/FAILURE)
 Main PID: 9528 (openvpn)
   Status: "Initialization Sequence Completed"
    Tasks: 1 (limit: 1152)
   CGroup: /<email address hidden>
           └─9528 /usr/sbin/openvpn --daemon ovpn-server --status /run/openvpn/server.status 10

May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: GID set to nogroup
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: UID set to nobody
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: MULTI: multi_init called, r=256 v=256
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: IFCONFIG POOL: base=10.8.0.4 size=62,
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: IFCONFIG POOL LIST
May 26 17:11:33 openvpn-reload-bionic ovpn-server[9528]: Initialization Sequence Completed
May 26 17:11:49 openvpn-reload-bionic systemd[1]: Reloading OpenVPN connection to server.
May 26 17:11:49 openvpn-reload-bionic kill[9548]: kill: (9528): Operation not permitted
May 26 17:11:49 openvpn-reload-bionic systemd[1]: <email address hidden>: Control process exite
May 26 17:11:49 openvpn-reload-bionic systemd[1]: Reload failed for OpenVPN connection to serve

This error (it returns 1) might confuse users. However, as you can see above, even when the reload command fails the service itself remains active and running. The same happens if we add the '+', but the reload command succeeds (returns 0).

Anyway, I'll prepare a MP to drop the reload support in Groovy while we decide if it worth to SRU a "fix" for this to the supported releases.

Changed in openvpn (Ubuntu):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
status: Triaged → In Progress
importance: Undecided → Medium
Changed in openvpn (Ubuntu Bionic):
importance: Undecided → Medium
Changed in openvpn (Ubuntu Eoan):
importance: Undecided → Medium
Changed in openvpn (Ubuntu Focal):
importance: Undecided → Medium

After a discussion with a colleague I think we should leave it as is in the supported/stable releases (I already fixed it in Groovy, it should land in the archive soon). The rationale is: even if the reload command fails the service keeps active and running, the proposed solutions would not fix the problem but would hide it from users. Adding the CAP_KILL capability or the '+' prefix would allow the service to send the SIGHUP signal and make 'systemctl reload openvpn@<server>' return 0, however, it would still face some permission denied errors, making the reload fails silently. In short, with or without the proposed fixes the reload process will not succeed.

Simon Déziel (sdeziel) wrote :

I'd like to mention that `systemctl $action openvpn@$foo` is using the legacy systemd unit.

On 18.04 and newer, you are better off moving your config to /etc/openvpn/client or /etc/openvpn/server and use corresponding and modern systemd units: openvpn-client@$foo and openvpn-server@$foo

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.4.9-2ubuntu2

---------------
openvpn (2.4.9-2ubuntu2) groovy; urgency=medium

  * Drop reload support from systemd unit files (LP: #1868127)

 -- Lucas Kanashiro <email address hidden> Tue, 26 May 2020 19:04:33 -0300

Changed in openvpn (Ubuntu):
status: In Progress → Fix Released

I brought more people into this discussion and we decided to drop the reload support from the stable/supported releases since it is already broken and the restart does basically the same thing.

Changed in openvpn (Ubuntu Bionic):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in openvpn (Ubuntu Eoan):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in openvpn (Ubuntu Focal):
assignee: nobody → Lucas Kanashiro (lucaskanashiro)
Changed in openvpn (Ubuntu Bionic):
status: Triaged → In Progress
Changed in openvpn (Ubuntu Eoan):
status: Triaged → In Progress
Changed in openvpn (Ubuntu Focal):
status: Triaged → In Progress
description: updated
summary: - OpenVPN will not reload due to misconfigured .service file
+ [SRU] OpenVPN will not reload due to misconfigured .service file
description: updated
Robie Basak (racb) wrote :

Hi Lucas,

> ...we decided to drop the reload support from the stable/supported releases since it is already broken and the restart does basically the same thing.

I'm puzzled by this. What effect will dropping the reload support have?

Is it that it previously failed, and with this SRU it will continue to fail but for a different reason? If so, what's the benefit to users for landing this SRU?

Is it that it previously failed, and with this SRU it will now succeed but with a more appropriate behaviour? If so, please could you document this?

Hi Robie,

With the proposed change users will not get the error mentioned in the description while trying to reload the service and get a better error (reload not supported). The current status is:

1 - User tries to reload the service
2 - Reload fails
3 - User gets an error

The problem here is that the service keeps active and running, so some users might infer that everything is fine and the reload process might have worked. But actually it did not.

With the proposed change we will drop the reload support so users will know that it did not do what they were expecting. I hope this make sense for you.

Robie Basak (racb) wrote :

OK, thanks. Let's accept these into the proposed pockets, but defer updating users until there a more significant update needed - then this change can be bundled in.

tags: added: block-proposed-bionic block-proposed-eoan block-proposed-focal
Changed in openvpn (Ubuntu Focal):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-focal

Hello John, or anyone else affected,

Accepted openvpn into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.7-1ubuntu2.20.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvpn (Ubuntu Eoan):
status: In Progress → Fix Committed
tags: added: verification-needed-eoan
Robie Basak (racb) wrote :

Hello John, or anyone else affected,

Accepted openvpn into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.7-1ubuntu2.19.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in openvpn (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed-bionic
Robie Basak (racb) wrote :

Hello John, or anyone else affected,

Accepted openvpn into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/openvpn/2.4.4-2ubuntu1.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Anyone willing to do the verification work?

Simon Déziel (sdeziel) wrote :
Download full text (10.0 KiB)

Reproduction procedure followed for all verification:

1) initial problem:
 apt install --no-install-recommends -y openvpn

 touch /etc/openvpn/lp1868127.conf
 chmod 0600 /etc/openvpn/lp1868127.conf
 cat << EOF > /etc/openvpn/lp1868127.conf
 ...
 EOF
 systemctl start openvpn@lp1868127
 systemctl reload openvpn@lp1868127

 # observe reload failing

2) update:

 # deploy -proposed update
 echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" >> /etc/apt/sources.list
 apt update && apt install openvpn

3) confirm fix:
 systemctl reload openvpn@lp1868127

This last step must print:

> Failed to reload <email address hidden>: Job type reload is not applicable for unit <email address hidden>.

Focal:

1) initial problem:

root@focal-openvpn-lp1868127:~# systemctl reload openvpn@lp1868127
Job for <email address hidden> failed.
See "systemctl status <email address hidden>" and "journalctl -xe" for details.

2) update:

root@focal-openvpn-lp1868127:~# apt update && apt install openvpn
Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease
Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal-proposed InRelease [265 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [310 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [116 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [28.9 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [7,664 B]
Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [142 kB]
Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [71.7 kB]
Get:11 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [11.6 kB]
Get:12 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [3,892 B]
Get:13 http://archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages [37.3 kB]
Get:14 http://archive.ubuntu.com/ubuntu focal-proposed/main Translation-en [17.6 kB]
Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [147 kB]
Get:16 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [51.1 kB]
Get:17 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [28.9 kB]
Get:18 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [7,664 B]
Get:19 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [42.8 kB]
Get:20 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [22.6 kB]
Fetched 1,530 kB in 1s (1,311 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  resolvconf openvpn-systemd-resolved easy-rsa
The following packages will be upgraded:
  openvpn
1 upgraded, 0 newly installed, 0 to...

tags: added: verification-done verification-done-bionic verification-done-eoan verification-done-focal
removed: verification-needed verification-needed-bionic verification-needed-eoan verification-needed-focal

The verification of the Stable Release Update for openvpn has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package openvpn - 2.4.7-1ubuntu2.19.10.1

---------------
openvpn (2.4.7-1ubuntu2.19.10.1) eoan; urgency=medium

  * Drop reload support from systemd unit files (LP: #1868127)

 -- Lucas Kanashiro <email address hidden> Wed, 27 May 2020 19:25:32 -0300

Changed in openvpn (Ubuntu Eoan):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers