Support snaps inside of lxd containers

[Eric Snow]

I tried following the instructions on snapcraft.io and got a failure. See the output below. I've also attached the relevant output from running "journalctl -xe".

uname: Linux 3.19.0-65-generic x86_64
release: Ubuntu 16.04
package: snapd 2.11+0.16.04

Notably, I'm running this in an LXD container (version: 2.0.0.rc9).

---------------------

$ sudo snap install hello-world
64.75 MB / 64.75 MB [==========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] 100.00 % 2.85 MB/s

error: cannot perform the following tasks:
- Mount snap "ubuntu-core" (122) ([start snap-ubuntu\x2dcore-122.mount] failed with exit status 1: Job for snap-ubuntu\x2dcore-122.mount failed. See "systemctl status "snap-ubuntu\\x2dcore-122.mount"" and "journalctl -xe" for details.
)
$ ls -la /snap
total 4K
drwxr-xr-x 3 root root 4096 Aug 8 17:49 ubuntu-core
$ ls -la /snap/ubuntu-core/
total 4K
drwxr-xr-x 2 root root 4096 Aug 8 17:49 122
$ ls -la /snap/ubuntu-core/122/
total 0K
$ systemctl status "snap-ubuntu\\x2dcore-122.mount"
● snap-ubuntu\x2dcore-122.mount - Mount unit for ubuntu-core
   Loaded: loaded (/etc/systemd/system/snap-ubuntu\x2dcore-122.mount; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Mon 2016-08-08 17:49:36 UTC; 6min ago
    Where: /snap/ubuntu-core/122
     What: /var/lib/snapd/snaps/ubuntu-core_122.snap
  Process: 31781 ExecMount=/bin/mount /var/lib/snapd/snaps/ubuntu-core_122.snap /snap/ubuntu-core/122 -t squashfs (code=exited, status=32)

Aug 08 17:49:35 my-host systemd[1]: Mounting Mount unit for ubuntu-core...
Aug 08 17:49:35 my-host mount[31781]: mount: /snap/ubuntu-core/122: mount failed: Unknown error -1
Aug 08 17:49:36 my-host systemd[1]: snap-ubuntu\x2dcore-122.mount: Mount process exited, code=exited status=32
Aug 08 17:49:36 my-host systemd[1]: Failed to mount Mount unit for ubuntu-core.
Aug 08 17:49:36 my-host systemd[1]: snap-ubuntu\x2dcore-122.mount: Unit entered failed state.

[Eric Snow (ericsnowcurrently)]

I found this: https://lists.ubuntu.com/archives/snapcraft/2016-June/000128.html

[Chad Smith (chad.smith)]

Hit a comparable issue with my own built snap:

systemctl status "snap-ubuntu\\x2dcore-122.mount"
● snap-ubuntu\x2dcore-122.mount - Mount unit for ubuntu-core
   Loaded: loaded (/etc/systemd/system/snap-ubuntu\x2dcore-122.mount; enabled; v
   Active: failed (Result: exit-code) since Mon 2016-08-08 21:35:08 UTC; 59s ago
    Where: /snap/ubuntu-core/122
     What: /var/lib/snapd/snaps/ubuntu-core_122.snap
  Process: 2932 ExecMount=/bin/mount /var/lib/snapd/snaps/ubuntu-core_122.snap /

Aug 08 21:35:08 lxd-xenial2 systemd[1]: Mounting Mount unit for ubuntu-core...
Aug 08 21:35:08 lxd-xenial2 mount[2932]: mount: /snap/ubuntu-core/122: mount fai
Aug 08 21:35:08 lxd-xenial2 systemd[1]: snap-ubuntu\x2dcore-122.mount: Mount pro
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to mount Mount unit for ubuntu-co
Aug 08 21:35:08 lxd-xenial2 systemd[1]: snap-ubuntu\x2dcore-122.mount: Unit ente
root@lxd-xenial2:~/src/snaps# journalctl -xe
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to reset devices.list on /system.
Aug 08 21:35:08 lxd-xenial2 mount[2932]: mount: /snap/ubuntu-core/122: mount fai
Aug 08 21:35:08 lxd-xenial2 systemd[1]: snap-ubuntu\x2dcore-122.mount: Mount pro
Aug 08 21:35:08 lxd-xenial2 systemd[1]: Failed to mount Mount unit for ubuntu-co
-- Subject: Unit snap-ubuntu\x2dcore-122.mount has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit snap-ubuntu\x2dcore-122.mount has failed.
--
-- The result is failed.
Aug 08 21:35:08 lxd-xenial2 systemd[1]: snap-ubuntu\x2dcore-122.mount: Unit ente
Aug 08 21:35:08 lxd-xenial2 /usr/lib/snapd/snapd[88]: task.go:250: DEBUG: 2016-0
Aug 08 21:35:08 lxd-xenial2 /usr/lib/snapd/snapd[88]: taskrunner.go:238: DEBUG:
Aug 08 21:35:54 lxd-xenial2 dhclient[230]: DHCPREQUEST of 10.160.159.48 on eth0
Aug 08 21:35:54 lxd-xenial2 dhclient[230]: DHCPACK of 10.160.159.48 from 10.160.
Aug 08 21:35:54 lxd-xenial2 dhclient[230]: bound to 10.160.159.48 -- renewal in

[Tyler Hicks (tyhicks)]

Ubuntu Security is working on AppArmor changes to allow, among other things, snap-confine to load AppArmor profiles inside of LXD containers.

[Tyler Hicks (tyhicks)]

I believe that LXD changes will be needed, as well. I'll let Stéphane set the status and importance as he sees fit.

[Stéphane Graber (stgraber)]

Yeah, LXD will need updating but we're already tracking this work and actually have it all ready to merge as soon as your team delivers a working kernel.

[Stéphane Graber (stgraber)]

We can't merge it before then as there's currently no way for us to detect a broken kernel vs a good kernel, so merging this work would effectively break all LXD users on a kernel that pretends to support namespacing and stacking, such as the 16.04 release kernel.

[Mark Shuttleworth (sabdfl)]

Thanks guys, can't wait to see this in action :)

Mark

[Tyler Hicks (tyhicks)]

John has gotten all of the AppArmor kernel changes merged into the Yakkety kernel and my apparmor userspace upload is making its way through the autopkgtests.

[Tyler Hicks (tyhicks)]

apparmor 2.10.95-4ubuntu5 has landed in Yakkety.

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.8.0-19.21

---------------
linux (4.8.0-19.21) yakkety; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1629057

  * 4.8.0 kernels do not complete boot process on VM (LP: #1627198)
    - [Config] CONFIG_HARDENED_USERCOPY_PAGESPAN=n

  * mount-image-callback cannot mount partitioned disk image (LP: #1628336)
    - SAUCE: nbd: Only delay uevent until connected

  * Support snaps inside of lxd containers (LP: #1611078)
    - apparmor: add interface to be able to grab loaded policy
    - securityfs: update interface to allow inode_ops, and setup from vfs fns
    - apparmor: refactor aa_prepare_ns into prepare_ns and create_ns routines
    - apparmor: add __aa_find_ns fn
    - apparmor: add mkdir/rmdir interface to manage policy namespaces
    - apparmor: fix oops in pivot_root mediation
    - apparmor: fix warning that fn build_pivotroot discards const
    - apparmor: add interface to advertise status of current task stacking
    - apparmor: update policy permissions to consider ns being viewed/managed
    - apparmor: add per ns policy management interface
    - apparmor: bump domain stacking version to 1.2

  * linux-image-extra-4.8.0-17-generic does not provide many sound card modules
    (LP: #1628523)
    - [Config] CONFIG_ZONE_DMA=y for generic

  * Yakkety - disable ARCH_ZX (LP: #1628503)
    - [Config] armhf: disable ARCH_ZX

  * Enable switchdev config parameter for Yakkety (LP: #1628241)
    - [Config] CONFIG_NET_SWITCHDEV=y for amd64/arm64

  * Ubuntu 16.10 kernel v4.8: Installation failing on Habanero with Shiner card
    (LP: #1628009)
    - firmware: Update bnx2x to 7.13.1.0

  * vNIC driver missing in 4.8 kernel package (LP: #1628187)
    - [Config] Enable CONFIG_IBMVNIC=m

  * Yakkety - armhf: MFD_TPS65217 and REGULATOR_TPS65217 are boot essential
    (LP: #1628112)
    - [Config] armhf: MFD_TPS65217=y && REGULATOR_TPS65217=y

  * Miscellaneous Ubuntu changes
    - Rebase to v4.8-rc8
    - [Config] skip Ubuntu-4.8.0-18.20
    - [Config] missing modules in armhf/s390x

  * Miscellaneous Ubuntu changes
    - rebase to v4.8-rc8

 -- Leann Ogasawara <email address hidden> Sun, 25 Sep 2016 12:13:35 -0700

[Stéphane Graber (stgraber)]

Marking this bug fix released as all the bits we wanted done here have been done.

We still have a separate bug open for the dependency on squashfuse and its SRU to xenial.

[Seth Forshee (sforshee)]

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

[dt9394 (dknyvice)]

The fix only for Yakkety? I still have the same error on Xenial with proposed enabled.

[dt9394 (dknyvice)]

NVM, I will download and compile the kernel and test it this week. Let see how it go.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dt9394 (dknyvice)]

I tested using latest xenial proposed kernel with latest apparmor utils. The problem still there.

[Seth Forshee (sforshee)]

Based on feedback from @jjohansen there will be follow-up patches to fix the problems, but the patches already applied should be kept and do not need to be reverted.

[John Johansen (jjohansen)]

note: that for xenial there are several pieces that must land as different SRUs. Just using the xenial SRU kernel is not sufficient. There is an apparmor userspace SRU that is required, and squashfuse sru ...

[Launchpad Janitor (janitor)]

This bug was fixed in the package linux - 4.4.0-47.68

---------------
linux (4.4.0-47.68) xenial; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1636941

  * Add a driver for Amazon Elastic Network Adapters (ENA) (LP: #1635721)
    - lib/bitmap.c: conversion routines to/from u32 array
    - net: ethtool: add new ETHTOOL_xLINKSETTINGS API
    - net: ena: Add a driver for Amazon Elastic Network Adapters (ENA)
    - [config] enable CONFIG_ENA_ETHERNET=m (Amazon ENA driver)

  * unexpectedly large memory usage of mounted snaps (LP: #1636847)
    - [Config] switch squashfs to single threaded decode

 -- Kamal Mostafa <email address hidden> Wed, 26 Oct 2016 10:47:55 -0700

[Tyler Hicks (tyhicks)]

Xenial's apparmor was fixed with package version 2.10.95-0ubuntu2.5

[Andrew Wilkins (axwalk)]

Is this supposed to work on Yakkety now? I've just tried, and it fails in the same way for me. I'm using:

  lxd 2.6.2-0ubuntu1~ubuntu16.10.1~ppa1
  lxd-client 2.6.2-0ubuntu1~ubuntu16.10.1~ppa1
  apparmor 2.10.95-4ubuntu5.1
  snapd 2.17.1+16.10

Linux 4.8.0-28-generic x86_64

[Stéphane Graber (stgraber)]

Did you install squashfuse in your container?

[Andrew Wilkins (axwalk)]

> Did you install squashfuse in your container?

Thanks, that was the missing link. Works after installing squashfuse.

For anyone else wondering, instructions are in the description of lp:1630789.

[dt9394 (dknyvice)]

Testing on Xenial with 4.8 Ubuntu kernel.

In container,
ubuntu@test:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial

ubuntu@test:~$ dpkg -l | grep -i 'apparmor\|snap\|squash'
ii apparmor 2.10.95-0ubuntu2.5 amd64 user-space parser utility for AppArmor
ii libapparmor-perl 2.10.95-0ubuntu2.5 amd64 AppArmor library Perl bindings
ii libapparmor1:amd64 2.10.95-0ubuntu2.5 amd64 changehat AppArmor library
ii snap-confine 2.20.1ubuntu1 amd64 Support executable to apply confinement for snappy apps
ii snapd 2.20.1ubuntu1 amd64 Tool to interact with Ubuntu Core Snappy.
ii squashfs-tools 1:4.3-3ubuntu2 amd64 Tool to create and append to squashfs filesystems
ii squashfuse 0.1.100-0ubuntu1~ubuntu16.04.1 amd64 FUSE filesystem to mount squashfs archives
ii ubuntu-core-launcher 2.20.1ubuntu1 amd64 Launcher for ubuntu-core (snappy) apps

sudo snap install hello-world
error: cannot perform the following tasks:
- Setup snap "core" (714) security profiles (cannot setup udev for snap "core": cannot reload udev rules: exit status 2

[Nicholas Skaggs (nskaggs)]

I can confirm this works on xenial after installing squashfuse.

root@clean-lark:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.1 LTS
Release: 16.04
Codename: xenial
root@clean-lark:~# uname -a
Linux clean-lark 4.4.0-63-generic #84-Ubuntu SMP Wed Feb 1 17:20:32 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
root@clean-lark:~# dpkg -l | grep -i 'apparmor\|snap\|squash'
ii apparmor 2.10.95-0ubuntu2.5 amd64 user-space parser utility for AppArmor
ii libapparmor-perl 2.10.95-0ubuntu2.5 amd64 AppArmor library Perl bindings
ii libapparmor1:amd64 2.10.95-0ubuntu2.5 amd64 changehat AppArmor library
ii snap-confine 2.21 amd64 Support executable to apply confinement for snappy apps
ii snapd 2.21 amd64 Tool to interact with Ubuntu Core Snappy.
ii squashfs-tools 1:4.3-3ubuntu2 amd64 Tool to create and append to squashfs filesystems
ii ubuntu-core-launcher 2.21 amd64 Launcher for ubuntu-core (snappy) apps
root@clean-lark:~# snap list
Name Version Rev Developer Notes
core 16.04.1 888 canonical -
hello-world 6.3 27 canonical -

[dt9394 (dknyvice)]

The latest version of xenial kernel, apparmor , lxd are now have the fixes to run snap in lxd container. If it fail, please try to install squashfuse or disable privileged mode.

[Caio Begotti (caio1982)]

Is there a regression here? Launching a Yakkety 16.10 LXD container fails to install the core snap, fusermount cannot do its magic:

root@e:~# snap install core
error: cannot perform the following tasks:
- Mount snap "core" (1577) ([start snap-core-1577.mount] failed with exit status 1: Job for snap-core-1577.mount failed.
See "systemctl status snap-core-1577.mount" and "journalctl -xe" for details.
)

● snap-core-1577.mount - Mount unit for core
   Loaded: loaded (/etc/systemd/system/snap-core-1577.mount; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2017-04-05 15:42:15 UTC; 20s ago
    Where: /snap/core/1577
     What: /var/lib/snapd/snaps/core_1577.snap
  Process: 605 ExecMount=/bin/mount /var/lib/snapd/snaps/core_1577.snap /snap/core/1577 -t fuse.squashfuse -o ro,allow_other (code=exited, status=1/FA

Apr 05 15:42:15 e systemd[1]: Mounting Mount unit for core...
Apr 05 15:42:15 e mount[605]: fusermount: mount failed: Operation not permitted
Apr 05 15:42:15 e systemd[1]: snap-core-1577.mount: Mount process exited, code=exited status=1
Apr 05 15:42:15 e systemd[1]: Failed to mount Mount unit for core.
Apr 05 15:42:15 e systemd[1]: snap-core-1577.mount: Unit entered failed state.

[Greg Lutostanski (lutostag)]

And in some cases squashfuse may not have the proper dep of fuse. And you may need to manually:

apt install fuse # in addition to squashfuse

<stgraber> looks like squashfuse is missing a dependency on "fuse"
<stgraber> which is part of the official Ubuntu images but not in the community images
<stgraber> so if you use "lxc launch ubuntu:16.04" it'll work (after you install squashfuse)
<stgraber> but if you use "images:ubuntu/xenial" you'll need to install "fuse" and "squashfuse"

[dt9394 (dknyvice)]

Latest snap-confine seem break the lxd snap function. It used to work until recent update.

snap-confine 2.22.6
lxd 2.12-0ubuntu3~ubuntu16.04.1~ppa1
linux-image-4.4.0-72-generic 4.4.0-72.93

Apr 18 15:33:22 snapbox audit[15919]: AVC apparmor="DENIED" operation="file_inherit" namespace="root//lxd-devbox_<var-lib-lxd>" profile="/usr/lib/snapd/snap-confine" name="/dev/tty" pid=15919 comm="snap-confine" requested_mask="wr" denied_mask="wr" fsuid=265536 ouid=0
Apr 18 15:33:22 snapbox audit[15919]: SYSCALL arch=c000003e syscall=59 success=yes exit=0 a0=c820161b00 a1=c820194150 a2=c82008bb20 a3=0 items=2 ppid=15917 pid=15919 auid=4294967295 uid=265536 gid=265536 euid=265536 suid=265536 fsuid=265536 egid=265536 sgid=265536 fsgid=265536 tty=(none) ses=4294967295 comm="snap-confine" exe="/usr/lib/snapd/snap-confine" key=(null)
Apr 18 15:33:22 snapbox audit: BPRM_FCAPS fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000003ffdfcffff old_pi=0000000000000000 old_pe=0000003ffdfcffff new_pp=0000003ffdfcffff new_pi=0000000000000000 new_pe=0000003ffdfcffff
Apr 18 15:33:22 snapbox audit: EXECVE argc=4 a0="/usr/lib/snapd/snap-confine" a1="snap.hello-world.hello-world" a2="/usr/lib/snapd/snap-exec" a3="hello-world"
Apr 18 15:33:22 snapbox audit: CWD cwd="/home/ubuntu"
Apr 18 15:33:22 snapbox audit: PATH item=0 name="/usr/lib/snapd/snap-confine" inode=27527378 dev=08:02 mode=0104755 ouid=265536 ogid=265536 rdev=00:00 nametype=NORMAL
Apr 18 15:33:22 snapbox audit: PATH item=1 name="/lib64/ld-linux-x86-64.so.2" inode=19678033 dev=08:02 mode=0100755 ouid=265536 ogid=265536 rdev=00:00 nametype=NORMAL

[dt9394 (dknyvice)]

I notice container upgrade from 14.04 to 16.04 do not have /lib/modules directory. It caused snap install error when /lib/modules do not exist in the container.

- Run configure hook of "core" snap if present (run hook "configure": cannot perform operation: mount --rbind /lib/modules /tmp/snap.rootfs_5c56PD//lib/modules: No such file or directory)

solution: mkdir /lib/modules

proposed solution: create /lib/modules when lxd container detected.

[Stéphane Graber (stgraber)]

No, the solution is that snapd shouldn't assume that /lib/modules exist and just not attempt to bind-mount it if it's missing.

Systems that don't have kernels installed (like containers) shouldn't have /lib/modules at all.

[Jacek Nykis (jacekn)]

I am still unable to run snaps inside lxd containers. I've just tested on an Ubuntu 16.04.3 LTS host:

$ uname -a
Linux hp 4.10.0-33-generic #37~16.04.1-Ubuntu SMP Fri Aug 11 14:07:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ lxc version
2.0.10
$ lxc launch ubuntu:16.04 test
Creating test
Starting test
$ lxc exec test -- apt update
<redacted for readability>
$ lxc exec test -- apt dist-upgrade -y
<redacted for readability>
$ lxc exec test -- apt install squashfuse -y
<redacted for readability>
$ lxc exec test -- snap install hello
error: cannot communicate with server: Post http://localhost/v2/snaps/hello: dial unix /run/snapd.socket: connect: connection refused
$ lxc exec test -- systemctl status snapd
● snapd.service - Snappy daemon
   Loaded: loaded (/lib/systemd/system/snapd.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Wed 2017-08-30 22:39:39 UTC; 35s ago
 Main PID: 2017 (code=exited, status=201/NICE)

Aug 30 22:39:39 test systemd[1]: snapd.service: Unit entered failed state.
Aug 30 22:39:39 test systemd[1]: snapd.service: Failed with result 'exit-code'.
Aug 30 22:39:39 test systemd[1]: snapd.service: Service hold-off time over, scheduling restart.
Aug 30 22:39:39 test systemd[1]: Stopped Snappy daemon.
Aug 30 22:39:39 test systemd[1]: snapd.service: Start request repeated too quickly.
Aug 30 22:39:39 test systemd[1]: Failed to start Snappy daemon.
$ lxc exec test -- snap version
snap 2.26.10
snapd unavailable
series -
$

[Haw Loeung (hloeung)]

Sounds like it might be LP:1709536