strongswan (charon) is rejected by apparmor to read /proc/<PID>/fd

Bug #1786250 reported by fermulator on 2018-08-09
22
This bug affects 3 people
Affects Status Importance Assigned to Milestone
strongswan (Ubuntu)
Undecided
Andreas Hasenack

Bug Description

[Impact]

strongswan needs to read from /proc/<PID>/fd
In some configurations, when apparmor blocks access, strongswan fails to set up properly.

[Test Case]

Unable to set up a reliable test case.
Tried setting up a VPN between two hosts, restarting strongswan, taking the eth device down and up, setting and removing routes, rebooting. Nothing seemed to trigger it.

[Regression Potential]

This is an expansion of permissions, which may increase the attack surface of strongswan.

[Original Description]

Used to work fine in Ubuntu 16.04 LTS, and Ubuntu 17.10.

ii strongswan 5.6.2-1ubuntu2 all IPsec VPN solution metapackage

A while ago I upgrade to 18.04 LTS and had consistent issues with strongswan ipsec connectivity VPN.

BASELINE INFO:

$ sudo ipsec statusall
Status of IKE charon daemon (strongSwan 5.6.2, Linux 4.15.0-29-generic, x86_64):
  uptime: 13 seconds, since Aug 09 09:27:35 2018
  malloc: sbrk 3268608, mmap 532480, used 1280560, free 1988048
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 0
  loaded plugins: charon test-vectors unbound ldap pkcs11 tpm aesni aes rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey dnscert ipseckey pem openssl gcrypt af-alg fips-prf gmp curve25519 agent chapoly xcbc cmac hmac ctr ccm gcm ntru bliss curl soup mysql sqlite attr kernel-netlink resolve socket-default connmark farp stroke vici updown eap-identity eap-sim eap-sim-pcsc eap-aka eap-aka-3gpp2 eap-simaka-pseudonym eap-simaka-reauth eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap eap-tnc xauth-generic xauth-eap xauth-pam xauth-noauth tnc-tnccs tnccs-20 tnccs-11 tnccs-dynamic dhcp whitelist lookip error-notify certexpire led radattr addrblock unity counters
Listening IP addresses:
  1.0.0.6
  192.168.130.9
  192.168.140.17
  192.168.130.14
  192.168.140.2
  172.17.0.1
  192.168.122.1
Connections:
  <SITE_SNIPPED>primary: %any...<SITE_SNIPPED>primary.<SNIPPED>.com IKEv2, dpddelay=30s
  <SITE_SNIPPED>primary: local: [<USER_SNIPPED>] uses EAP_MSCHAPV2 authentication
  <SITE_SNIPPED>primary: remote: [OU=Domain Control Validated, CN=<SNIPPED>.com] uses public key authentication
  <SITE_SNIPPED>primary: child: 192.168.140.0/24 === 192.168.128.0/17 10.0.0.0/8 172.16.0.0/12 TUNNEL, dpdaction=clear
<SITE_SNIPPED>secondary: %any...<SITE_SNIPPED>secondary.<SNIPPED>.com IKEv2, dpddelay=30s
<SITE_SNIPPED>secondary: local: [<USER_SNIPPED>] uses EAP_MSCHAPV2 authentication
<SITE_SNIPPED>secondary: remote: [OU=Domain Control Validated, CN=<SNIPPED>.com] uses public key authentication
<SITE_SNIPPED>secondary: child: 192.168.130.0/24 === 192.168.128.0/17 10.0.0.0/8 172.16.0.0/12 TUNNEL, dpdaction=clear
Routed Connections:
<SITE_SNIPPED>secondary{2}: ROUTED, TUNNEL, reqid 2
<SITE_SNIPPED>secondary{2}: 192.168.130.0/24 === 10.0.0.0/8 172.16.0.0/12 192.168.128.0/17
  <SITE_SNIPPED>primary{1}: ROUTED, TUNNEL, reqid 1
  <SITE_SNIPPED>primary{1}: 192.168.140.0/24 === 10.0.0.0/8 172.16.0.0/12 192.168.128.0/17
Security Associations (0 up, 0 connecting):
  none

Then we do:

```
 sudo ipsec up <CONNECTION_NAME>

... all the goods happen ...

but near the end:

IKE_SA <CONNECTION_NAME>[1] established between 1.0.0.6[<USER_SNIPPED>]...64.7.137.180[OU=Domain Control Validated, CN=<SNIPPED_HOST>.com]
scheduling reauthentication in 56358s
maximum IKE_SA lifetime 56538s
installing DNS server 192.168.194.20 via resolvconf
installing DNS server 192.168.196.20 via resolvconf
<<HANGS FOREVER>>
```

the DNSes are successfully added to resolvconf (/etc/resolv.conf) - however the resolution doesn't work, and no routes work with the VPN.

After a fresh reboot, this works.
No end of ipsec/strongswan service restarts gets the system out of this "stuck state";

--
Typical workflow (reproduction notes)
 1. fresh boot
 2. VPN connections fine
 3. work work work
 4. disconnect VPN
 5. use system for personal use (or don't)
 6. suspend system overnight
 7. resume system morning
 8. VPN BROKEN as noted above
--

Digging more, I see these errors in dmesg
```
...
[34218.436021] audit: type=1400 audit(1533821247.602:169): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/21179/fd/" pid=21179 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[34368.429799] audit: type=1400 audit(1533821397.596:170): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/22483/fd/" pid=22483 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[34368.437049] audit: type=1400 audit(1533821397.604:171): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/22493/fd/" pid=22493 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[34380.630335] audit: type=1400 audit(1533821409.796:172): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/22656/fd/" pid=22656 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[34395.681889] audit: type=1400 audit(1533821424.847:173): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/22882/fd/" pid=22882 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
[34395.688730] audit: type=1400 audit(1533821424.855:174): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/22888/fd/" pid=22888 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
...
```

Does this have anything to do with why the connection is hanging? I have no idea.

Tried this:

$ sudo /etc/init.d/apparmor status
● apparmor.service - AppArmor initialization
   Loaded: loaded (/lib/systemd/system/apparmor.service; enabled; vendor preset: enabled)
   Active: active (exited) since Thu 2018-08-09 09:19:09 EDT; 15min ago
     Docs: man:apparmor(7)
           http://wiki.apparmor.net/
  Process: 9518 ExecStop=/etc/init.d/apparmor stop (code=exited, status=0/SUCCESS)
  Process: 14731 ExecStart=/etc/init.d/apparmor start (code=exited, status=0/SUCCESS)
 Main PID: 14731 (code=exited, status=0/SUCCESS)

Aug 09 09:19:06 fermmy systemd[1]: Starting AppArmor initialization...
Aug 09 09:19:06 fermmy apparmor[14731]: * Starting AppArmor profiles
Aug 09 09:19:06 fermmy apparmor[14731]: Skipping profile in /etc/apparmor.d/disable: usr.bin.firefox
Aug 09 09:19:06 fermmy apparmor[14731]: Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd
Aug 09 09:19:09 fermmy apparmor[14731]: ...done.
Aug 09 09:19:09 fermmy systemd[1]: Started AppArmor initialization.

$ sudo /etc/init.d/apparmor stop
[ ok ] Stopping apparmor (via systemctl): apparmor.service.

REPEAT TEST

 * restart strongswan
 * unroute all connections manually (sudo ipsec unroute <CONNECTION>)

_wtf_, apparmor is STILL rejecting it! (even though it's stopped?)

[34756.774786] audit: type=1400 audit(1533821785.933:177): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/26204/fd/" pid=26204 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

--
NO ACCESS to /proc ;/

$ cd /etc/apparmor.d
$ grep -rins charon * | grep proc
(EMPTY)

--
need to unload charon profile

$ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.ipsec.charon

but STILL, rejecting

[35206.129530] audit: type=1400 audit(1533822235.279:249): apparmor="DENIED" operation="open" profile="/usr/lib/ipsec/charon" name="/proc/30951/fd/" pid=30951 comm="charon" requested_mask="r" denied_mask="r" fsuid=0 ouid=0

Related branches

fermulator (fermulator) wrote :

Also probably worth including the current ipsec.charon profile contents (even though it's disabled now ...)

summary: - strongswan (charon) is rejected by apparmor to read /proc/<PID/fd
+ strongswan (charon) is rejected by apparmor to read /proc/<PID>/fd
fermulator (fermulator) wrote :
fermulator (fermulator) wrote :

while ipsec is still, here are the contents of the /proc/<PID>/fd it's trying to access
```
$ sudo ls -al /proc/3014/fd/
total 0
dr-x------ 2 root root 0 Aug 9 09:51 .
dr-xr-xr-x 9 root root 0 Aug 9 09:51 ..
lr-x------ 1 root root 64 Aug 9 09:51 0 -> 'pipe:[2972727]'
l-wx------ 1 root root 64 Aug 9 09:51 1 -> 'pipe:[2972728]'
lrwx------ 1 root root 64 Aug 9 09:51 10 -> 'socket:[2961426]'
l-wx------ 1 root root 64 Aug 9 09:51 2 -> 'pipe:[2972728]'
```

fermulator (fermulator) wrote :

repeated with more care to ensure profiles are actually unloaded

running this twice, confirms profiles are now not loaded

$ for profile in $(find . | egrep "charon|ipsec" | grep -v local); do sudo apparmor_parser -R /etc/apparmor.d/$profile; done
apparmor_parser: Unable to remove "/usr/lib/ipsec/lookip". Profile doesn't exist
apparmor_parser: Unable to remove "/usr/sbin/charon-systemd". Profile doesn't exist
apparmor_parser: Unable to remove "/usr/lib/ipsec/stroke". Profile doesn't exist
apparmor_parser: Unable to remove "/usr/lib/ipsec/charon". Profile doesn't exist

and, the aa-status confirms
$ sudo aa-status | egrep "ipsec|charon"
(EMPTY)

---

RETRY

 - ffs, connection STILL hangs, but these rejected charon messages in dmesg are no longer happening (so maybe those are a legit bug/issue with the profile to be fixed, but a red-herring to my primary issue)

fermulator (fermulator) wrote :

Submitted a "fork" bug report for the connection hang issue (https://bugs.launchpad.net/ubuntu/+source/strongswan/+bug/1786261), let this bug report stay for the charon apparmor profile issue.

Hi,
could you add to the apparmor profile of charon this line
   @{PROC}/@{pid}/fd/ r,
Then reload it via:
   sudo apparmor_parser -r /etc/apparmor.d/usr.lib.ipsec.charon

While I never have heard of charon needing this, if the above works you could add it for youself as a config and I could make it part of future packages.

If the above makes those messages disappear but shows new apparmor denies afterwards let me know.

fermulator (fermulator) wrote :

Did this:

```
$ grep include /etc/apparmor.d/usr.lib.ipsec.charon | grep local
  include <local/usr.lib.ipsec.charon>

$ cat /etc/apparmor.d/local/usr.lib.ipsec.charon
# Site-specific additions and overrides for usr.lib.ipsec.charon.
# For more details, please see /etc/apparmor.d/local/README.
#
# https://bugs.launchpad.net/ubuntu/+source/strongswan/+bug/1786250
@{PROC}/@{pid}/fd/ r,
```

Then reloaded
```
$ sudo apparmor_parser -r /etc/apparmor.d/usr.lib.ipsec.charon
```

Now more complaints in dmesg.

I assume, as you have found completely disabling it before - your'd actual hang isn't gone by that - right?

Changed in strongswan (Ubuntu):
status: New → Triaged
tags: added: server-next

TODO:
add
  @{PROC}/@{pid}/fd/ r,
to the charon apparmor profile

tags: added: bitesize
fermulator (fermulator) on 2018-08-20
Changed in strongswan (Ubuntu):
assignee: nobody → fermulator (fermulator)
fermulator (fermulator) wrote :

Patched:

$ git status
On branch allow_charon_apparmor_read_proc_fd_LP_#1786250

commit d0ec74d30d6742d34b3dc72113bbc933c608fffa (HEAD -> allow_charon_apparmor_read_proc_fd_LP_#1786250)
Author: (SNIP) <fermulator>
Date: Mon Aug 20 09:40:38 2018 -0400

    As per LP #1786250, user noted audit failures in system log
    against charon trying to read its own list of file descriptors
    in /proc/<pid>/fd/.

    We are uncertain when/why this started, however it is not
    unreasonable for a process to attempt to read its own fd's,
    so allow by extending the apparmor profile for charon.

    References:
    http://manpages.ubuntu.com/manpages/bionic/en/man5/apparmor.d.5.html
    https://linux.die.net/man/5/proc

@kstenerud - please review, integrate, fixup, test, build, ... the usual things.
@fermulator - kstenerud will take care to carry your fix into Ubuntu.

The attachment "proposal for fix to charon apparmor profile" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Karl Stenerud (kstenerud) wrote :

@fermulator - Thanks for the merge proposal! We're getting started on it but in the meantime, could you help with a couple of things?

1. We are trying to come up with a simple test case, but if you have one already (config files, etc), that would help a lot!

2. Could you add a commit to your branch called "changelog" which adds an entry to debian/changelog? You can make one by running the dch command from the top level of the strongswan repo.

The message part can be something like this:

  * debian/usr.lib.ipsec.charon: allow self to read file descriptors.
    (LP #1786250)

Andreas Hasenack (ahasenack) wrote :

Karl, could you add this comment in the MP instead please?

description: updated
Changed in strongswan (Ubuntu):
assignee: fermulator (fermulator) → Karl Stenerud (kstenerud)
description: updated

Taking over the cleanup as Beta Freeze is close and the change is too easy to miss it and have much more work later on.

Changed in strongswan (Ubuntu):
assignee: Karl Stenerud (kstenerud) →  Christian Ehrhardt  (paelzer)

@Fermulator - I polished, your upload and will prepare it.
Without a real hard issue other than the error dmesg I'd not try to SRu to former releases.
Debian should be affected just as much, would you mind reporting a bug there as well so that they acn also pick the change at some point?

If you do so please report back the bug or PR link here so we can track, if you are unable to do so let me know and I can do that on the next merge (but original reporter is preferred in case there are questions about the case).

Note there also seems to be a build/test issue (bug 1794224) due to the new toolchain that needs to be fixed along that.

FTBFS resolved (actually is a LP infra issue).
Successfully tested the PPA.

fermulator (fermulator) wrote :

(where in Debian/upstream should I report to?)

fermulator (fermulator) wrote :

(also note, since I went mia not expecting to need to track, Christian took over and did the cleanup - his merge is https://code.launchpad.net/~paelzer/ubuntu/+source/strongswan/+git/strongswan/+merge/355589)

The best Debian entry page IMHO is the tracker [1].
That would lead you to bugs [2].
And that would make you aware of bug reporting [3].

TL;DR: a mail to <email address hidden> with:
<your subject>
Package: strongswan
Version: 5.7.0-1
<your text>

[1]: https://tracker.debian.org/pkg/strongswan
[2]: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=strongswan
[3]: https://www.debian.org/Bugs/Reporting

fermulator (fermulator) wrote :

(Does the general debian package though care about the Ubuntu apparmor rules?)

Yes they have adopted our rules.
Here the Debian file you'd want to change:
https://salsa.debian.org/debian/strongswan/blob/debian/master/debian/usr.lib.ipsec.charon

Boris Tomici (xbrs1) wrote :

Hello guys,

Today we have updated our testing environment to Ubuntu 18.04 and also updated strongSwan from 5.2 to 5.6.2.

After spending all day with migration of configuration we encountered this problem :/.

Any fast way to fix it because our testing team is stuck.

Best regards,
Boris

Hi Boris,
the real fix is on the way, but it will need to complete in 18.10 first (where currently we have Beta Freeze) and then also needs some time to SRU into Bionic.

For now you can modify your config file in /etc/apparmor.d/usr.lib.ipsec.charon and add the line
 @{PROC}/@{pid}/fd/ r,
That should resolve the access to the PID being an issue.

Boris Tomici (xbrs1) wrote :
Download full text (4.2 KiB)

Hi Christian,

Did as you said and restarted apparmor but for me is the same.

Connection is established but no traffic goes thru.

root@vsrv-bicab-2u:/home/VPN# cat /etc/apparmor.d/usr.lib.ipsec.charon
# ------------------------------------------------------------------
#
# Copyright (C) 2016 Canonical Ltd.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of version 2 of the GNU General Public
# License published by the Free Software Foundation.
#
# Author: Jonathan Davies <email address hidden>
# Ryan Harper <email address hidden>
#
# ------------------------------------------------------------------

#include <tunables/global>

/usr/lib/ipsec/charon flags=(attach_disconnected) {
  #include <abstractions/base>
  #include <abstractions/nameservice>
  #include <abstractions/authentication>
  #include <abstractions/openssl>
  #include <abstractions/p11-kit>

  capability ipc_lock,
  capability net_admin,
  capability net_raw,

  # allow priv dropping (LP: #1333655)
  capability chown,
  capability setgid,
  capability setuid,

  # libcharon-extra-plugins: xauth-pam
  capability audit_write,

  # libstrongswan-standard-plugins: agent
  capability dac_override,

  capability net_admin,
  capability net_raw,

  network,
  network raw,

  /bin/dash rmPUx,

  # libchron-extra-plugins: kernel-libipsec
  /dev/net/tun rw,

  /etc/ipsec.conf r,
  /etc/ipsec.secrets r,
  /etc/ipsec.*.secrets r,
  /etc/ipsec.d/ r,
  /etc/ipsec.d/** r,
  /etc/ipsec.d/crls/* rw,
  /etc/opensc/opensc.conf r,
  /etc/strongswan.conf r,
  /etc/strongswan.d/ r,
  /etc/strongswan.d/** r,
  /etc/tnc_config r,

  /proc/sys/net/core/xfrm_acq_expires w,

  /run/charon.* rw,
  /run/pcscd/pcscd.comm rw,

  /usr/lib/ipsec/charon rmix,
  /usr/lib/ipsec/imcvs/ r,
  /usr/lib/ipsec/imcvs/** rm,

  /usr/lib/*/opensc-pkcs11.so rm,

  /var/lib/strongswan/* r,

  @{PROC}/@{pid}/fd/ r,

  # Site-specific additions and overrides. See local/README for details.
  #include <local/usr.lib.ipsec.charon>
}

root@vsrv-bicab-2u:/home/VPN# date
Thu Sep 27 15:28:13 UTC 2018

Sep 27 15:28:46 vsrv-bicab-2u charon: 12[IKE] IKE_SA l2tp-ikev2-rw-ah[1] established between 192.168.231.2[C=DE, O=KDLabs, CN=vpnclientAHL2TP@kdlabs]...192.168.231.1[192.168.231.1]
Sep 27 15:28:46 vsrv-bicab-2u charon: 12[IKE] scheduling reauthentication in 9729s
Sep 27 15:28:46 vsrv-bicab-2u charon: 12[IKE] maximum IKE_SA lifetime 10269s
Sep 27 15:28:46 vsrv-bicab-2u charon: 12[IKE] adding DNS server failed
Sep 27 15:28:46 vsrv-bicab-2u charon: 12[IKE] adding DNS server failed
Sep 27 15:28:46 vsrv-bicab-2u charon: 12[CFG] handling INTERNAL_IP4_DNS attribute failed
Sep 27 15:28:46 vsrv-bicab-2u kernel: [10627.234397] audit: type=1400 audit(1538062126.282:80): apparmor="DENIED" operation="unlink" profile="/usr/lib/ipsec/charon" name="/etc/resolv.conf" pid=4190 comm="charon" requested_mask="d" denied_mask="d" fsuid=0 ouid=0
Sep 27 15:28:46 vsrv-bicab-2u kernel: [10627.234408] audit: type=1400 audit(1538062126.2...

Read more...

TL;DR - not the same bug, please open a new one

Hi,
well I only explained how to avoid the issue of the self FD access.
This is what this bug is about.

And your report doesn't have that anymore.
Please open a new bug for your issue.

For the things I see in what you posted seems to be about resolve.conf updates driven by/through strongswan. That really is a different issue.
The line:
#include <abstractions/nameservice>
Should cover resolve.conf actions, but we'd have to understand exactly how your strongswan is configured triggering this issue and then consider what/where to add apparmor rules for.

Please just use your last post for a new bug and feel free to copy&paste my reply as an answer.

From there add your config details especially related to resolve.conf updating and we can start thinking about it.

Boris Tomici (xbrs1) wrote :

Hi Christian,

As those are fresh installed machines, I didn't check that there was a typo in resolve.conf i have solve it and it works now.

Bests,
Boris

Changed in strongswan (Ubuntu):
assignee:  Christian Ehrhardt  (paelzer) → Andreas Hasenack (ahasenack)
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package strongswan - 5.6.3-1ubuntu4

---------------
strongswan (5.6.3-1ubuntu4) cosmic; urgency=medium

  * d/usr.lib.ipsec.charon: allow reading of own FDs (LP: #1786250)
    Thanks to Matt Callaghan.

 -- Andreas Hasenack <email address hidden> Thu, 04 Oct 2018 10:34:01 -0300

Changed in strongswan (Ubuntu):
status: In Progress → Fix Released
Yichen Wang (yicwang) wrote :

I've did couple of tests. On iOS 12, I do see this error from /var/log/messages, but connection can still be established with xl2tpd, and things are working fine. For Windows 10 clients, seems like it is blocking and connection cannot be established...

Actually, do we have the plan to back ported this version to Bionic?

Andreas Hasenack (ahasenack) wrote :

@yicwang, did you update the apparmor profile as per this bug for your tests? The diff can be seen at https://code.launchpad.net/~ahasenack/ubuntu/+source/strongswan/+git/strongswan/+merge/356135.

Make that change, then run this command:

sudo apparmor_parser -r -T -W /etc/apparmor.d/usr.lib.ipsec.charon

If that fixes the problem for you, and the apparmor DENIED messages about this disappear from the logs, then we can start an SRU for bionic.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers