lxd snap fails to install w/apparmor "permission denied" error

[lee]

Got the notice when updating today that LXD is moving to snap-based distribution and the PPAs are being killed, so I tried to follow the instructions and do a quick n simple switchover to snap LXD. Unfortunately, this is what happens:

--
# snap install lxd && lxd.migrate
error: cannot perform the following tasks:
- Run configure hook of "lxd" snap if present (run hook "configure":
-----
cannot update snap namespace: cannot open /proc/self/cmdline: permission denied
snap-update-ns failed with code 1
-----)
--

Failure is the same if I try --edge instead of default. Syslog has some more info in it—it looks like apparmor is wigging out, but I'm not smart enough to understand why:

--
Nov 19 11:58:05 ultracarl snapd[2845]: 2017/11/19 11:58:05.159257 api.go:957: Installing snap "lxd" revision unset
Nov 19 11:58:07 ultracarl systemd[1]: Reloading.
Nov 19 11:58:07 ultracarl systemd[1]: Reloading.
Nov 19 11:58:07 ultracarl systemd[1]: Mounting Mount unit for lxd...
Nov 19 11:58:07 ultracarl systemd[1]: Mounted Mount unit for lxd.
Nov 19 11:58:07 ultracarl kernel: [177757.660819] kauditd_printk_skb: 4 callbacks suppressed
Nov 19 11:58:07 ultracarl kernel: [177757.660822] audit: type=1400 audit(1511114287.622:157): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.benchmark" pid=2194553 comm="apparmor_parser"
Nov 19 11:58:07 ultracarl kernel: [177757.840825] audit: type=1400 audit(1511114287.802:158): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.check-kernel" pid=2194555 comm="apparmor_parser"
Nov 19 11:58:07 ultracarl kernel: [177757.973259] audit: type=1400 audit(1511114287.934:159): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.daemon" pid=2194557 comm="apparmor_parser"
Nov 19 11:58:08 ultracarl kernel: [177758.173255] audit: type=1400 audit(1511114288.134:160): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.hook.configure" pid=2194559 comm="apparmor_parser"
Nov 19 11:58:08 ultracarl kernel: [177758.361893] audit: type=1400 audit(1511114288.322:161): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.lxc" pid=2194561 comm="apparmor_parser"
Nov 19 11:58:08 ultracarl kernel: [177758.501580] audit: type=1400 audit(1511114288.462:162): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.lxd" pid=2194563 comm="apparmor_parser"
Nov 19 11:58:08 ultracarl kernel: [177758.669990] audit: type=1400 audit(1511114288.630:163): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="snap.lxd.migrate" pid=2194565 comm="apparmor_parser"
Nov 19 11:58:08 ultracarl systemd[1]: Reloading.
Nov 19 11:58:08 ultracarl systemd[1]: Reloading.
Nov 19 11:58:09 ultracarl systemd[1]: Started Service for snap application lxd.daemon.
Nov 19 11:58:09 ultracarl kernel: [177759.303107] audit: type=1400 audit(1511114289.266:164): apparmor="DENIED" operation="open" profile="/snap/core/3440/usr/lib/snapd/snap-confine//snap_update_ns" name="/proc/2194647/cmdline" pid=2194647 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 19 11:58:09 ultracarl lxd.daemon[2194632]: cannot update snap namespace: cannot open /proc/self/cmdline: permission denied
Nov 19 11:58:09 ultracarl lxd.daemon[2194632]: snap-update-ns failed with code 1
Nov 19 11:58:09 ultracarl systemd[1]: snap.lxd.daemon.service: Main process exited, code=exited, status=1/FAILURE
Nov 19 11:58:09 ultracarl kernel: [177759.506684] audit: type=1400 audit(1511114289.470:165): apparmor="DENIED" operation="open" profile="/snap/core/3440/usr/lib/snapd/snap-confine//snap_update_ns" name="/proc/2194664/cmdline" pid=2194664 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 19 11:58:09 ultracarl systemd[1]: Reloading.
Nov 19 11:58:09 ultracarl kernel: [177759.711214] audit: type=1400 audit(1511114289.674:166): apparmor="DENIED" operation="open" profile="/snap/core/3440/usr/lib/snapd/snap-confine//snap_update_ns" name="/proc/2194684/cmdline" pid=2194684 comm="5" requested_mask="r" denied_mask="r" fsuid=0 ouid=0
Nov 19 11:58:09 ultracarl lxd.daemon[2194655]: cannot update snap namespace: cannot open /proc/self/cmdline: permission denied
Nov 19 11:58:09 ultracarl lxd.daemon[2194655]: snap-update-ns failed with code 1
Nov 19 11:58:09 ultracarl systemd[1]: snap.lxd.daemon.service: Control process exited, code=exited status=1
Nov 19 11:58:09 ultracarl systemd[1]: Stopped Service for snap application lxd.daemon.
Nov 19 11:58:09 ultracarl systemd[1]: snap.lxd.daemon.service: Unit entered failed state.
Nov 19 11:58:09 ultracarl systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Nov 19 11:58:09 ultracarl systemd[1]: Reloading.
Nov 19 11:58:10 ultracarl snapd[2845]: 2017/11/19 11:58:10.353498 handlers.go:310: Reported install problem for "lxd" as 343543bc-cd53-11e7-9298-fa163e8d4bab OOPSID
Nov 19 11:58:10 ultracarl systemd[1]: Unmounted Mount unit for lxd.
Nov 19 11:58:10 ultracarl systemd[1]: Reloading.
--

I contacted Stéphane Graber, one of the LXD developers, and he suggested that this might be due to a regression in snapd 2.29, so I wanted to file this report in case that happens to be the case.

Quick extra bits of info:

--
# uname -ar
Linux ultracarl 4.10.0-38-generic #42~16.04.1-Ubuntu SMP Tue Oct 10 16:32:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
--

--
# snap version
snap 2.29.3
snapd 2.29.3
series 16
ubuntu 16.04
kernel 4.10.0-38-generic
--

[Stéphane Graber (stgraber)]

I've seen this happen on 3 of my own systems (out of 10 that I was moving to the snap).
On all of them, rebooting the system was enough to correct the issue and have the LXD snap install properly.

This looks like some odd kernel/snap-update-ns interaction in snapd. Since my systems were production systems, I've rebooted them all and so don't have any currently broken system to perform additional tests on.

[Michael Vogt (mvo)]

Adding zyga who is most familiar with apparmor issues

[Seyeong Kim (seyeongkim)]

I could reproduce this issue with core revision 3604 ( not in lxd )

and I found apparmor patch for this. it worked.

https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/1717714

but with core revision 3748 i can't reproduce this.

I'm going to try this with lxd

[Seyeong Kim (seyeongkim)]

below commit [1] avoid this issue.

[1]
commit a1fa5e9781ab19b7c1c89ce651772c120a2d353a
Author: James Henstridge <email address hidden>
Date: Fri Nov 10 17:58:44 2017 +0800

    cmd: use a preinit_array function rather than parsing /proc/self/cmdline

[Michael Vogt (mvo)]

Thanks @xtrusia !