apparmor denial using ptmx char device

[Pedro Coca]

[Impact]

snap-confine would refuse to work on an older kernel running on an Nvidia Tegra X1 board. This was traced to a bug in older version of apparmor there that required directory-like syntax for /dev/pts/ptmx (with a trailing slash).

This bug is fixed by adding an apparmor rule, identical to the normal rule, with an extra slash. Older kernels will use the new rule while current kernels will just ignore it.

[Test Case]

On an Nvidia Tegra X1 board, running 3.10.96 snap-confine should no longer fail to start. On Ubuntu Xenial (all architectures) there should be no perceived change.

Snap-confine is carefully tested with a battery of spread tests that can be found here: https://github.com/snapcore/snap-confine/blob/master/spread-tests/

The test cases are ran automatically for each pull request and for each final release.

All those tests were executed successfully for this release. As a simple test case consider running any snap (any at all, including hello-world).

[Regression Potential]

 * Regression potential is minimal as the fix simply adds another apparmor rule that grants additional permissions that are only picked up by old buggy kernels.

* The fix was tested on Ubuntu via spread.

[Other Info]

* This bug is a part of a major SRU that brings snap-confine in Ubuntu 16.04 in line with the current upstream release 1.0.41.

* This bug was included in an earlier SRU and is now fixed in Ubuntu. I am updating the template here to ensure that the process is fully documented from 1.0.38 all the way up to the current upstream release 1.0.41.

* snap-confine is technically an integral part of snapd which has an SRU exception and is allowed to introduce new features and take advantage of accelerated procedure. For more information see https://wiki.ubuntu.com/SnapdUpdates

== # Pre-SRU bug description follows # ==

- Finding issues running snaps (hello-world).
- Same issue even installing with --devmode. Even running the snap binary as root
- Using a custom kernel, this is on an Nvidia Tegra X1 custom board.

=====================================

ubuntu@localhost:~$ hello-world.echo plop
unable to mount '/dev/pts/ptmx'->'/dev/ptmx'. errmsg: Permission denied
ubuntu@localhost:~$ sudo hello-world.echo plop
unable to mount '/dev/pts/ptmx'->'/dev/ptmx'. errmsg: Permission denied

dmesg shows:
=====================================

[ 302.838046] type=1400 audit(1455208371.989:16): apparmor="DENIED"
operation="mount" info="failed mntpnt match" error=-13 parent=911
profile="/usr/bin/ubuntu-core-launcher" name="/dev/ptmx/" pid=912
comm="ubuntu-core-lau" srcname="/dev/pts/ptmx/" flags="rw, bind"
[ 308.080449] type=1400 audit(1455208377.229:17): apparmor="DENIED"
operation="mount" info="failed mntpnt match" error=-13 parent=914
profile="/usr/bin/ubuntu-core-launcher" name="/dev/ptmx/" pid=915
comm="ubuntu-core-lau" srcname="/dev/pts/ptmx/" flags="rw, bind"

This is with the "hello-world" snap installed with "snap install"

Output of an ls over the device file:
=====================================

ubuntu@localhost:~$ ls -lR /dev/ptmx /dev/pts
crw-rw-rw- 1 root tty 5, 2 Feb 11 16:28 /dev/ptmx

/dev/pts:
total 0
c--------- 1 root root 5, 2 Jan 1 1970 ptmx

[Jamie Strandboge (jdstrand)]

Thanks for the bug report. We've discussed this on IRC and in a private email thread. I'll summarize here:

"Basically the launcher wants to mount /dev/pts/ptmx on /dev/ptmx and there are
apparmor rules in the launcher's apparmor profile that allow that. However, the
denial is for 'name="/dev/ptmx/"' -- notice the trailing '/' in /dev/ptmx/ --
this is not allowed by the launcher's profile and /dev/ptmx should be a file,
not a directory. From 'man pts': "The file /dev/ptmx is a character file with
major number 5 and minor number 2, ...".

There appears to be a problem with your system, perhaps udev rules. How are your
/dev/ptmx and /dev/pts being setup before any snaps are run (including snaps
with daemons started by systemd)? Are you using a custom kernel, gadget and/or
os snap?"

Can you provide the following:
1. uninstall all app snaps, reboot then provide the output of 'ls -lR /dev/ptmx /dev/pts'
2. 'snap install hello-world' and then run 'hello-world.echo foo', then provide the output of 'ls -lR /dev/ptmx /dev/pts'
3. What kernel snap are you using? If custom, can you attach it?
4. What gadget snap are you using? If custom, can you attach it?
5. What device are you targeting? (eg, amd64 VM, rpi2, rpi3, etc)

Thanks!

[Jamie Strandboge (jdstrand)]

I see you answered '5' already: Nvidia Tegra X1 custom board. You mentioned you have a custom kernel, can you attach it? On IRC, I believe you said you had a custom gadget snap, can you attach it?

[Patrick Boettcher (patrick-boettcher)]

I worked with Yann on this issue today and can answer some of your questions:

1) there are no snaps installed other than

  - ubuntu-core
  - kernel (a modified raspberry-pi-kernel snap where modules/image/uboot.env has been changed)
  - hello-world

2)
There is no difference in the output of 'ls -lR /dev/ptmx /dev/pts' between before and after running hello-world.echo.

5) Tegra X1 Kernel version is 3.10.96+ (NVidia Tegra X1 + Parrot's drivers)

Running

  sudo mount -o bind,rw /dev/pts/ptmx /dev/ptmx

works.

The problem seems to be in the kernel part of apparmor adding the trailing slashes on this bind mount paths. Are there any custom patches applied to stock kernels before the can work with ubuntu-core?

[Yann Sionneau (yann-sionneau)]

I'll try to complete Patrick's answer here:

1°)
root@localhost:~# snap list
Name Version Rev Developer
canonical-pi2 3.2 6 canonical
canonical-pi2-linux 4.4.0-1011-raspi2+20160518.15-59 13 canonical
ubuntu-core 16.04+20160518.15-47 116 canonical

<REBOOT>
ubuntu@localhost:~$ ls -lR /dev/ptmx /dev/pts
crw-rw-rw- 1 root tty 5, 2 May 24 09:12 /dev/ptmx

/dev/pts:
total 0
c--------- 1 root root 5, 2 Jan 1 1970 ptmx

2°)
root@localhost:~# snap install hello-world

Name Version Rev Developer
hello-world 6.0 25 canonical
root@localhost:~# hello-world.echo foo
unable to mount '/dev/pts/ptmx'->'/dev/ptmx'. errmsg: Permission denied
root@localhost:~# ls -lR /dev/ptmx /dev/pts
crw-rw-rw- 1 root tty 5, 2 Feb 11 16:28 /dev/ptmx

/dev/pts:
total 0
c--------- 1 root root 5, 2 Jan 1 1970 ptmx

3°) I'm using the RPI2 kernel snap as a basis, I unsquashfs'ed it, replaced the kernel to an arm64 Tegra X1 one, replaced the kernel modules and the firmwares.
4°) I'm using the unmodified rpi2 gadget snap
5°) I'm targeting a custom Tegra X1 board (armv8), I'm using arm64 kernel and armhf (32 bits) userspace.

the snap list :
root@localhost:~# snap list
Name Version Rev Developer
canonical-pi2 3.2 6 canonical
canonical-pi2-linux 4.4.0-1011-raspi2+20160518.15-59 13 canonical
hello-world 6.0 25 canonical
ubuntu-core 16.04+20160518.15-47 116 canonical

root@localhost:~# uname -a
Linux localhost.localdomain 3.10.96-g8a6a217 #2 SMP PREEMPT Wed May 18 11:23:45 CEST 2016 aarch64 aarch64 aarch64 GNU/Linux

To further explain how I produced the Snappy image :

I'm not able to generate a Snappy image right now using UDF due to this bug: https://bugs.launchpad.net/ubuntu/+source/goget-ubuntu-touch/+bug/1579767

So for now, I'm using a bit of a hackish technique to generate an image for my Tegra board.

I am using this script: https://github.com/fallen/snappy-tools/blob/master/paros_image/genimage.sh

This is basically generating an RPI2 image and then modifying the kernel snap to contain the correct kernel+modules+firmwares.
And same thing with the initramfs.

[Jamie Strandboge (jdstrand)]

You mentioned you are using this kernel: 3.10.96-g8a6a217. Are you using the upstream apparmor or have you backported the Ubuntu patches? If the latter, where did you get those patches from?

[Yann Sionneau (yann-sionneau)]

Hello Jamie,

We are using 3.10.96 from NVidia release, on top of that I applied the "AppArmor 2.4 compatibility patch" which adds this kconfig symbol : SECURITY_APPARMOR_COMPAT_24.

Do you mean I should cherry-pick the commits starting with "UBUNTU SAUCE" from there: http://kernel.ubuntu.com/git/ppisati/ubuntu-vivid.git/log/?h=snappy_v3.10 ?

Thanks!

[Yann Sionneau (yann-sionneau)]

I tried to rm -rf security/apparmor (which is what http://kernel.ubuntu.com/git/ppisati/ubuntu-vivid.git/commit/?h=snappy_v3.10&id=1268d90dcb4ad270eda64caba03ae95bd7cc2115 is doing) and then applying all the following "UBUNTU: SAUCE: " patches.

I still get the ptmx issue.
Note that the "UBUNTU: [Config]" patches are already applied on my config file.

[Tyler Hicks (tyhicks)]

Thanks for the update, Yann. I've built the snappy_v3.10 tagged kernel, installed it in a 16.04 VM, and can reproduce the issue that you're seeing. I will begin debugging the problem.

[Yann Sionneau (yann-sionneau)]

To summarize what I tried :
I tried two different kernels :

1°)
kernel NVIDIA 3.10.96
+
8a6a217 [DEV] Add AppArmor 2.4 compatibility patch

The compatibility patch is in https://launchpad.net/apparmor/2.10/2.10.1/+download/apparmor-2.10.1.tar.gz in the directory /apparmor-2.10.1/kernel-patches/3.10/

=> sudo snap install : OK
=> run a snap binary : /dev/ptmx issue : KO

2°)
kernel NVIDIA 3.10.96
+
53ebbe9 sched: move no_new_privs into new atomic flags
534accf UBUNTU: SAUCE: Revert: fix: only allow a single threaded process to ...
7560bbe UBUNTU: SAUCE: apparmor: 3.11 backport revert module/lsm: Have apparm 5265fc62
cc38feb UBUNTU: SAUCE: apparmor: 3.12 backport mtd: Move major number f83c3838
3ea2c13 UBUNTU: SAUCE: apparmor: 3.12 backport kvfree: reintroduce kvfree() and vfree() helper function
7410a9e UBUNTU: SAUCE: apparmor: backport setup base backport files
1c8c390 UBUNTU: SAUCE: (no-up) apparmor: Sync to apparmor3 - RC1 snapshot
1268d90 UBUNTU: SAUCE: (no-up) apparmor: remove security/apparmor directory

Those patches are those available over there : http://kernel.ubuntu.com/git/ppisati/ubuntu-vivid.git/log/?h=snappy_v3.10

=> sudo snap install : OK
=> run a snap binary : /dev/ptmx issue : KO

Any news on this issue? This is very blocking for us as we cannot run any snap at all.
Thanks!

[Jamie Strandboge (jdstrand)]

While the fixes are being prioritized and worked on, I can give you a potential workaround to unblock you. Can you:
1. cp /etc/apparmor.d/usr.bin.ubuntu-core-launcher $HOME
2. adjust $HOME/usr.bin.ubuntu-core-launcher:
Under this line:
mount options=(rw bind) /dev/pts/ptmx -> /dev/ptmx, # for bind mounting

add:
mount options=(rw bind) /dev/pts/ptmx -> /dev/ptmx/, # LP: #1584456

3. load the policy into the kernel:
$ sudo apparmor_parser -r $HOME/usr.bin.ubuntu-core-launcher

Please report back if this works for you. On reboot you will have to run the apparmor_parser command.

[Chad Miller (cmiller)]

@jds, the LHS and RHS need trailing slashes.

    mount options=(rw bind) /dev/pts/ptmx/ -> /dev/ptmx/, # for bind mounting, LP: #1584456

[Yann Sionneau (yann-sionneau)]

Hey Chad and Jamie,
Thanks for your help, in fact I've already found and used this kind of workaround, I was more searching for a fix that would allow me to use an unmodified ubuntu-core since it is neither expected nor a very pleasant development experience.

I can confirm this works (the solution with both trailing slashes).

I'm more interested in understanding and fixing the origin of the reason for this trailing slash being added (by the kernel?).

Thanks!

[Alexander Sack (asac)]

can confirm that the revised workaround with chad miller also works for me on a 3.10 kernel.

[Luke Williams (wililupy)]

Hello, from my bug (LP: 1605438) Seth asked what kernel I am running:

3.8.25 with the following patches applied:

0001-UBUNTU-SAUCE-AppArmor-basic-networking-rules.patch
0002-apparmor-Fix-quieting-of-audit-messages-for-network-.patch
0003-UBUNTU-SAUCE-apparmor-Add-the-ability-to-mediate-mou.patch
aufs.patch
driver-support-intel-igb-bcm54616-phy.patch

I'm assuming it is an AppArmor patch that I am missing, but the AppArmor source does not have a specific patch for 3.18, so I used the 3.12 patches only. If I need to apply a specific patch, please let me know.

Thanks.

[Luke Williams (wililupy)]

I mistyped my kernel I am using:
It is 3.18.25. I have to use this kernel since I have pre-built kernel modules that were built against this kernel that are proprietary.

[Shuduo Sang (sangshuduo)]

same issue happens with 4.4.0-1019-raspi2 kernel on RPi2.

[Jamie Strandboge (jdstrand)]

Tyler and I discussed this and feel that since this bug is understood and fixed in newer kernels that adding a workaround rule to the policy is fine at this point.

[Jamie Strandboge (jdstrand)]

https://github.com/snapcore/snap-confine/pull/101

[Yann Sionneau (yann-sionneau)]

Thanks!

[Martin Pitt (pitti)]

Hello Pedro, or anyone else affected,

Accepted snap-confine into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snap-confine/1.0.38-0ubuntu0.16.04.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Zygmunt Krynicki (zyga)]

Yann could you please verify that this bug is fixed by the package in xenial-proposed?

You can find more information about the process at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification

[Zygmunt Krynicki (zyga)]

Per agreement with jdstrand it is sufficient to verify that the new policy is a superset (that is, it allows to do more, not less) of the old policy. This prevents the possibility of regressions. Given that the original bug was reported on a non-common hardware/kernel combination this serves as a sufficient SRU verification.

As a part of the verification the apparmro profile from /etc/apparmor.d/usr.lib.snapd.snap-confine was copied before and after the proposed upgrade. The package upgraded successfully so the new profile was also successfully compiled and loaded into the kernel. Both profiles were compared and the new rule, containing the extra trailing slash, was present in the diff.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snap-confine - 1.0.38-0ubuntu0.16.04.10

---------------
snap-confine (1.0.38-0ubuntu0.16.04.10) xenial; urgency=medium

  * debian/usr.lib.snapd.snap-confine:
    - synchronize apparmor profile with upstream 1.0.40 release.
    (LP: #1597842, LP: #1615113, LP: #1584456)

snap-confine (1.0.38-0ubuntu0.16.04.9) xenial; urgency=medium

  * debian/patches/04_not_die_unknown_locations.patch:
    - move to /var/lib/snapd/void (with mode 0) if the current
      location cannot be preserved (LP: #1612684)

 -- Zygmunt Krynicki <email address hidden> Wed, 24 Aug 2016 20:31:12 +0200

[Andy Whitcroft (apw)]

The verification of the Stable Release Update for snap-confine has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.