[16.04, lxc] Failed to reset devices.list on ...

[TJ]

This seems to be identical to Bug #1413193 which affected 15.04 and was marked Fix Released.

On a 16.04 host with a 16.04 container:

$ lxc-create -t download -n vpn01 -- -d ubuntu -r xenial -a amd64
$ lxc-start -n vpn02 -d

the container syslog shows repeated instances of "Failed to reset devices.list on ..." and continues generating such messages:

$ grep devices.list .local/share/lxc/vpn01/rootfs/var/log/syslog

May 16 14:37:20 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/systemd-remount-fs.service: Operation not permitted
May 16 14:37:20 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/systemd-random-seed.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/resolvconf.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/getty-static.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/systemd-user-sessions.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp-runlevel.service: Operation not permitted
May 16 14:37:23 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/rc-local.service: Operation not permitted
May 16 14:52:51 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-clean.service: Operation not permitted
May 16 14:52:51 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/ondemand.service: Operation not permitted
May 16 15:29:30 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted
May 16 15:31:11 vpn01 systemd[1]: Failed to reset devices.list on /system.slice/console-getty.service: Operation not permitted

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Alvaro Uria (aluria)]

Linux 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:55:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I've created an Ubuntu Xenial LXC on top of an Ubuntu Xenial node and have run:

snap install hello-world

root@juju-machine-0-lxc-1:~# journalctl -xe
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/mdadm.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/atd.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/dev-lxc-tty3.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/system-getty.slice: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/cloud-init.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/proc-uptime.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/systemd-logind.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/dev-lxc-tty1.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/dev-mqueue.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/ufw.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/proc-stat.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/resolvconf.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/proc-diskstats.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/ondemand.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/proc-sysrq\x2dtrigger.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/dev-lxc-tty2.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/cloud-config.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 systemd[1]: Failed to reset devices.list on /system.slice/run-user-1000.mount: Operation not permitted
Jan 18 17:23:15 juju-machine-0-lxc-1 syste...

[James Troup (elmo)]

If you ensure your container (and host kernel) are up-to-date and install squashfuse in the container, this works.

[James Troup (elmo)]

Per stgraber, squashfuse is currently going through MIR, once it's in main, snapd will be updated to depend on it, making this work out of the box.

[JuanJo Ciarlante (jjo)]

I can't make it work even after manually installing squashfuse
(FYI lxc created by juju deploy cs:ubuntu --to lxc:1 )

root@juju-machine-1-lxc-14:~# uname -a
Linux juju-machine-1-lxc-14 4.8.0-34-generic #36~16.04.1-Ubuntu SMP Wed Dec 21 18:55:08 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

root@juju-machine-1-lxc-14:~# apt-cache policy squashfuse
squashfuse:
  Installed: 0.1.100-0ubuntu1~ubuntu16.04.1
  Candidate: 0.1.100-0ubuntu1~ubuntu16.04.1
  Version table:
 *** 0.1.100-0ubuntu1~ubuntu16.04.1 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/universe amd64 Packages
        100 /var/lib/dpkg/status

root@juju-machine-1-lxc-14:~# snap install hello-world
- Mount snap "core" (888) ([start snap-core-888.mount] failed with exit status 1: Job for snap-core-888.mount failed. See "systemctl status snap-core-888.mount" and "journalctl -xe" for details.

root@juju-machine-1-lxc-14:~# journalctl -xe
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /system.slice/snap-core-888.mount: Operation not permitted
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Mounting Mount unit for core...
-- Subject: Unit snap-core-888.mount has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit snap-core-888.mount has begun starting up.
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /init.scope: Operation not permitted
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /user.slice: Operation not permitted
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /system.slice/dbus.service: Operation not permitted
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /system.slice/ondemand.service: Operation not permitted
Feb 08 22:54:21 juju-machine-1-lxc-14 systemd[1]: Failed to reset devices.list on /system.slice/sys-kernel-debug.mount: Operation not permitted
[...]

[James Troup (elmo)]

Right, so this is lxc 1.x vs. lxd. Y'all should be using lxd, not lxc 1.x.

[Dan Streetman (ddstreet)]

please reopen if this is still an issue