normal users can't run snaps inside of LXD containers

[Tyler Hicks]

[Impact]

TBD

[Test Case]

Look below for a test case.

[Regression Potential]

TBD

[Other Info]

* snap-confine is technically an integral part of snapd which has an SRU exception and is allowed to introduce new features and take advantage of accelerated procedure. For more information see https://wiki.ubuntu.com/SnapdUpdates

== # Pre-SRU bug description follows # ==

The kernel (4.8.0-19.21), apparmor (2.10.95-4ubuntu5), and lxd (2.4-0ubuntu1) needed for running snaps inside of LXD containers (bug #1611078) have all landed in Yakkety. We should be able to install squashfuse and snapd 2.16+16.10 (from yakkety-proposed) and then run snaps inside of unprivileged LXD containers.

I have verified that it works well for the root user inside of the container but there are some issues when a normal user attempts to run a snap command.

# Create yakkety container named "yakkety"
tyhicks@host:~$ lxc launch ubuntu-daily:devel yakkety
Creating yakkety
Starting yakkety

# Enter the container, enable yakkety-proposed, update, install the dependencies
tyhicks@host:~$ lxc exec yakkety bash
root@yakkety:~# echo "deb http://archive.ubuntu.com/ubuntu/ \
yakkety-proposed restricted main multiverse universe" > \
/etc/apt/sources.list.d/proposed.list
root@yakkety:~# echo -e "Package: *\nPin: release a=yakkety-proposed\n\
Pin-Priority: 400" > /etc/apt/preferences.d/proposed-updates
root@yakkety:~# apt-get update && apt-get dist-upgrade -y
...
root@yakkety:~# apt-get install -y squashfuse snapd/yakkety-proposed
...

# Rebooting the container should not be needed but is done for completeness
root@yakkety:~# reboot
tyhicks@host:~$ lxc exec yakkety bash

# Install the hello-world snap
root@yakkety:~# snap install hello-world
hello-world (stable) 6.3 from 'canonical' installed

# Snap commands work fine as root inside the container but not as a normal user
root@yakkety:~# /snap/bin/hello-world.env
SNAP_USER_COMMON=/root/snap/hello-world/common
...
root@yakkety:~# su - ubuntu -c '/snap/bin/hello-world.env'
internal error, please report: running "hello-world.env" failed: open /snap/hello-world/27/meta/snap.yaml: permission denied

# The normal user can't access /snap/hello-world/27 because of some oddness with the
# dentry
root@yakkety:~# ls -al /snap/hello-world
total 8
drwxr-xr-x 3 root root 4096 Oct 5 21:09 .
drwxr-xr-x 5 root root 4096 Oct 5 21:09 ..
drwxrwxr-x 4 root root 0 Jul 11 21:20 27
lrwxrwxrwx 1 root root 2 Oct 5 21:09 current -> 27
root@yakkety:~# su - ubuntu -c 'ls -al /snap/hello-world'
ls: cannot access '/snap/hello-world/27': Permission denied
total 8
drwxr-xr-x 3 root root 4096 Oct 5 21:09 .
drwxr-xr-x 5 root root 4096 Oct 5 21:09 ..
d????????? ? ? ? ? ? 27
lrwxrwxrwx 1 root root 2 Oct 5 21:09 current -> 27

[Tyler Hicks (tyhicks)]

Here's the contents of the kernel ring buffer while running the following commands:

tyhicks@host:~$ lxc start yakkety
tyhicks@host:~$ lxc exec yakkety bash
root@yakkety:~# su - ubuntu -c '/snap/bin/hello-world.env'
internal error, please report: running "hello-world.env" failed: open /snap/hello-world/27/meta/snap.yaml: permission denied

Note that all of the output occurs while `lxc start yakkety` is running and none of the log entries stem from the command where hello-world.env is invoked.

[Tyler Hicks (tyhicks)]

It looks like the squashfuse mounts have the wrong mount parameters:

# grep /snap /proc/self/mountinfo
496 348 0:59 / /snap/hello-world/27 rw,relatime - fuse.squashfuse squashfuse rw,user_id=0,group_id=0
497 348 0:60 / /snap/ubuntu-core/423 rw,relatime - fuse.squashfuse squashfuse rw,user_id=0,group_id=0

The squashfuse man page mentions the following option:

     -o allow_other
              allow access by other users

I'm assuming that mounting with '-o allow_other' would allow normal users to run snap commands but we should think through whether or not this is safe to do.

Additionally, it looks incorrect that the 'rw' mount parameter is given. In a non-container, classic environment where squashfs is used, the 'ro' mount parameter is given:

$ grep /snap /proc/self/mountinfo
83 24 7:1 / /snap/hello-world/27 rw,relatime shared:32 - squashfs /dev/loop1 ro
82 24 7:0 / /snap/ubuntu-core/352 rw,relatime shared:33 - squashfs /dev/loop0 ro

[Tyler Hicks (tyhicks)]

Problem #1 is that the snaps are being mounted incorrectly when squashfuse is being used in place of squashfs:

root@yakkety:~# umount /snap/ubuntu-core/423/
root@yakkety:~# umount /snap/hello-world/27/
root@yakkety:~# squashfuse -o ro,allow_other /var/lib/snapd/snaps/ubuntu-core_423.snap /snap/ubuntu-core/423/
root@yakkety:~# squashfuse -o ro,allow_other /var/lib/snapd/snaps/hello-world_27.snap /snap/hello-world/27/
root@yakkety:~# grep /snap /proc/self/mountinfo
496 348 0:61 / /snap/hello-world/27 ro,nosuid,nodev,relatime - fuse.squashfuse squashfuse ro,user_id=0,group_id=0,allow_other
497 348 0:62 / /snap/ubuntu-core/423 ro,nosuid,nodev,relatime - fuse.squashfuse squashfuse ro,user_id=0,group_id=0,allow_other

This lets us get a little farther along:

root@yakkety:~# su - ubuntu -c '/snap/bin/hello-world.env'

There's no output because ubuntu-core-launcher (not snap-confine?) is segfaulting (possibly due to an AppArmor denial). Here is the relevant info from the logs:

[ 9087.695184] audit: type=1400 audit(1475710180.784:529): apparmor="DENIED" operation="open" namespace="root//lxd-yakkety_<var-lib-lxd>" profile="/usr/lib/snapd/snap-confine" name="/proc/644/mountinfo" pid=29474 comm="ubuntu-core-lau" requested_mask="r" denied_mask="r" fsuid=296608 ouid=0
[ 9087.695195] ubuntu-core-lau[29474]: segfault at 0 ip 00007f37440c1484 sp 00007ffcc6b2fa50 error 4 in libc-2.24.so[7f3744053000+1bd000]

[Tyler Hicks (tyhicks)]

Problem #2 is a bug in procfs is breaking the snap-confine AppArmor profile and causing the following rule to not work as intended:

  owner @{PROC}/*/mountinfo r,

procfs is incorrectly setting the uid and gid on its /proc/PID/* inodes that represent a setuid-container-root process. Here's how I came to that conclusion:

# Create a setuid-container-root program that doesn't quickly exit so you can poke around procfs
tyhicks@host:~$ container_root=$(sudo stat -c %u /var/lib/lxd/containers/yakkety/rootfs/)
tyhicks@host:~$ echo $container_root
296608
tyhicks@host:~$ sudo cp $(which sleep) /var/lib/lxd/containers/yakkety/rootfs/usr/bin/setuid-sleep
tyhicks@host:~$ sudo chown $container_root:$container_root \
  /var/lib/lxd/containers/yakkety/rootfs/usr/bin/setuid-sleep
tyhicks@host:~$ sudo chmod u+s /var/lib/lxd/containers/yakkety/rootfs/usr/bin/setuid-sleep

# Run the setuid-sleep program as an unprivileged user inside the container.
# The procfs inodes for that process are incorrectly owned by init_ns root
tyhicks@host:~$ lxc exec yakkety -- su - ubuntu -c 'setuid-sleep 5m'
# From another host terminal
tyhicks@host:~$ sudo stat -c %u /proc/$(pidof setuid-sleep)/mountinfo
0

# Run the setuid-sleep program as root inside the container.
# The procfs inodes for that process are incorrectly owned by container_ns root
tyhicks@host:~$ lxc exec yakkety -- setuid-sleep 5m
# From another host terminal
tyhicks@host:~$ sudo stat -c %u /proc/$(pidof setuid-sleep)/mountinfo
296608

This explains the AppArmor denial from comment #3 containing "fsuid=296608 ouid=0". The setuid-container-root snap-confine task is correctly running as fsuid 296608 (container_ns root) but the mountinfo inode is correctly assigned uid 0 (init_ns root). This causes the "owner" conditional on the AppArmor rule to not match and for snap-confine to segfault after it can't access /proc/self/mountinfo.

The procfs kernel bug needs to be fixed but, in the meantime, we can probably drop the "owner" conditional in the snap-confine profile.

[Tyler Hicks (tyhicks)]

Problem #3, the final problem, is due to a missing AppArmor rule needed when the following PR was merged:

  https://github.com/snapcore/snap-confine/pull/145

After fixing the squashfuse mounts, as mentioned in comment #3, and dropping the "owner" conditional, as mentioned in comment #4 (be sure to reload the AppArmor profile after that), we see the following:

root@yakkety:~# su - ubuntu -c '/snap/bin/hello-world.env'
cannot change apparmor hat of the support process for mount namespace capture. errmsg: Permission denied
support process for mount namespace capture exited abnormally

This AppArmor denial is logged:

[14428.623321] audit: type=1400 audit(1475715521.677:546): apparmor="DENIED" operation="open" namespace="root//lxd-yakkety_<var-lib-lxd>" profile="/usr/lib/snapd/snap-confine" name="/proc/977/attr/current" pid=908 comm="ubuntu-core-lau" requested_mask="w" denied_mask="w" fsuid=296608 ouid=0

That PR resulted in the following call chain:

  main() -> sc_main() -> sc_create_or_join_ns_group() -> aa_change_hat()

aa_change_hat() must write to /proc/PID/attr/current but that PR didn't add a rule to allow that file access.

Adding the '@{PROC}/[0-9]*/attr/current w,' rule and reloading the profile allows us to run the hello-world.env snap command as a regular user inside of an unprivileged user namespace:

root@yakkety:~# su - ubuntu -c '/snap/bin/hello-world.env'
XDG_SESSION_ID=c13

[Tyler Hicks (tyhicks)]

I made an unfortunate typo in the following sentence found in comment #4:

This explains the AppArmor denial from comment #3 containing "fsuid=296608 ouid=0". The setuid-container-root snap-confine task is correctly running as fsuid 296608 (container_ns root) but the mountinfo inode is correctly assigned uid 0 (init_ns root).

It should have read:

This explains the AppArmor denial from comment #3 containing "fsuid=296608 ouid=0". The setuid-container-root snap-confine task is correctly running as fsuid 296608 (container_ns root) but the mountinfo inode is *incorrectly* assigned uid 0 (init_ns root).

[Jamie Strandboge (jdstrand)]

1.0.43-0ubuntu1 uploaded to yakkety.

[Andy Whitcroft (apw)]

Hello Tyler, or anyone else affected,

Accepted snap-confine into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snap-confine/1.0.43-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Launchpad Janitor (janitor)]

This bug was fixed in the package snap-confine - 1.0.43-0ubuntu1

---------------
snap-confine (1.0.43-0ubuntu1) yakkety; urgency=medium

  * New upstream release (LP: #1630479, LP: #1630492, LP: #1628612)
  * debian/patches/lp1630789.patch: allow running snaps by non-root users in
    LXD containers (LP: #1630789)

 -- Jamie Strandboge <email address hidden> Thu, 06 Oct 2016 12:29:59 +0000

[Tyler Hicks (tyhicks)]

Pull request for snapd: https://github.com/snapcore/snapd/pull/2112

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.16+16.10ubuntu1

---------------
snapd (2.16+16.10ubuntu1) yakkety; urgency=medium

  * systemd/systemd.go, systemd/systemd_test.go: Correct the mount arguments
    when mounting with squashfuse (LP: #1630789)

 -- Tyler Hicks <email address hidden> Thu, 06 Oct 2016 18:49:40 +0000

[Leo Arias (elopio)]

This can't really be tested yet for snap-confine in xenial proposed because the required packages are not yet in xenial. I tried bringing packages from yakkety and yakkety-proposed, but that didn't work, it was just a long shot. It even seems there is still a PR in flight for snapd.

I checked the individual pull requests for snap-confine and they have been thoroughly reviewed, and after an hour running different tests I haven't found any regression. Because we have an emergency and need to speed up the landing of snap-confine, I'm going to mark this as verified. However once all the other pieces are in place in xenial, this needs some exploratory for snaps inside lxc.

Thanks Andy.

[Launchpad Janitor (janitor)]

This bug was fixed in the package snap-confine - 1.0.43-0ubuntu1~16.04.1

---------------
snap-confine (1.0.43-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  * Backport from 16.10 (LP: #1630040)

snap-confine (1.0.43-0ubuntu1) yakkety; urgency=medium

  * New upstream release (LP: #1630479, LP: #1630492, LP: #1628612)
  * debian/patches/lp1630789.patch: allow running snaps by non-root users in
    LXD containers (LP: #1630789)

snap-confine (1.0.42-0ubuntu3) yakkety; urgency=medium

  * allow snap-confine to mount on /dev/pts/ptmx for LXD with /dev/ptmx
    symlink

snap-confine (1.0.42-0ubuntu2) yakkety; urgency=medium

  * add mmap to AppArmor policy for snap-confine for running snap-confine
    under LXD on 4.8 kernels

snap-confine (1.0.42-0ubuntu1) yakkety; urgency=medium

  * New upstream release
  * Drop patch skip-nsfs-magic-tests-on-old-kernels.patch (applied upstream)

snap-confine (1.0.41-0ubuntu2) yakkety; urgency=medium

  * add skip-nsfs-magic-tests-on-old-kernels.patch to disable NSFS tests on
    kernels older than 3.19 (LP: #1625565)

snap-confine (1.0.41-0ubuntu1) yakkety; urgency=medium

  * New upstream release, full list of issues is available at
    https://launchpad.net/snap-confine/+milestone/1.0.41
  * Drop all patches (included upstream).
  * Add version to apparmor run-time dependency.

snap-confine (1.0.40-1) unstable; urgency=medium

  * New upstream release, full list of issues is available at
    https://launchpad.net/snap-confine/+milestone/1.0.40
  * Drop apparmor profile from the debian/ directory and install it straight
    from upstream package. This is now automatically consistent with package
    configuration prefix.
  * Drop patch: prctl-compatibility.patch(applied upstream)
  * Add directory /var/lib/snapd/void to snap-confine
  * Add patch: 0001-Don-t-shellcheck-files-spread-prepare-script.patch that
    fixes make check due to a mistake upstream.
  * Add patch: 0001-Stop-using-deprecated-readdir_r.patch (LP: #1615615)

snap-confine (1.0.39-1) unstable; urgency=medium

  * New upstream release.
  * Remove d/patches/01_lp1606277.patch, applied upstream.

snap-confine (1.0.38-3) unstable; urgency=medium

  * debian/patches/prctl-compatibility.patch: add shadow definitions for
    compatibility with older kernel headers.
  * drop build-dependency on shellcheck, which is not used at build time
    and doesn't exist in trusty.
  * make ubuntu-core-launcher "arch:any" to workaround an issue in
    rm_conffile which does not deal with changing architectures
  * fix log-observer interface regression (LP: #1606277)

snap-confine (1.0.38-2) unstable; urgency=medium

  * Fix invocations of rm_conffile.
  * Update d/usr.lib.snapd.snap-confine to the latest upstream version to
    ensure content-sharing fully works.

snap-confine (1.0.38-1) unstable; urgency=medium

  * New upstream release.

 -- Jamie Strandboge <email address hidden> Thu, 06 Oct 2016 14:51:26 +0000

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for snap-confine has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Andy Whitcroft (apw)]

Hello Tyler, or anyone else affected,

Accepted snap-confine into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snap-confine/1.0.44-0ubuntu1~16.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andy Whitcroft (apw)]

Hello Tyler, or anyone else affected,

Accepted snapcraft into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snapcraft/1.0.44-0ubuntu1~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andy Whitcroft (apw)]

Hello Tyler, or anyone else affected,

Accepted snap-confine into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/snap-confine/1.0.44-0ubuntu1~16.04 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andreas Hasenack (ahasenack)]

Nowadays xenial-updates has an ever higher version of snap-confine: 2.21

[Nicholas Skaggs (nskaggs)]

Yakkety still has 1.0.43.

[Zygmunt Krynicki (zyga)]

This bug was fixed while snap-confine was a separate package. I'm marking the snappy task as fix-released.