systemd does not boot in a container

[Dimitri John Ledkov]

Opening against cloud-init for now, but ultimately might end up as bug-fixes / srus against some other packages in trusty.

[Scott Moser (smoser)]

Well, this was opened as a trusty issue, and in my memory it was only on trusty, but here is an attempt that fails on trusty or utopic host. I know that when we did this, there was no systemd-sysv package, so my attempt was more to invoke /lib/systemd/systemd as the init comand.

$ sudo apt-get update -q && sudo apt-get install -qy lxc </dev/null
$ sudo lxc-create -t ubuntu-cloud -n source-utopic-amd64 -- \
    --release=utopic --arch=amd64 --stream=daily
$ sudo lxc-clone -o source-utopic-amd64 -n u1
$ sudo lxc-start -n u1

## logged in as ubuntu:ubuntu

% sudo apt-get update &&
   sudo apt-get install -qy systemd-sysv </dev/null && sync

$ sudo lxc-stop --kill -n u1
$ sudo lxc-start -n u1
systemd 208 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX +IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
Detected virtualization 'lxc'.

Welcome to Ubuntu Utopic Unicorn (development branch)!

Failed to insert module 'autofs4'
Set hostname to <u1>.
/etc/mtab is not a symlink or not pointing to /proc/self/mounts. This is not supported anymore. Please make sure to replace this file by a symlink to avoid incorrect or misleading mount(8) output.
No control group support available, not creating root group.
[/lib/systemd/system/friendly-recovery.service:14] Executable path is not absolute, ignoring: dmesg --console-off
Cannot add dependency job for unit systemd-vconsole-setup.service, ignoring: Unit systemd-vconsole-setup.service failed to load: No such file or directory.
Cannot add dependency job for unit display-manager.service, ignoring: Unit display-manager.service failed to load: No such file or directory.
[ OK ] Listening on Syslog Socket.
[ OK ] Reached target Remote File Systems (Pre).
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Listening on Delayed Shutdown Socket.
Failed to open /dev/autofs: No such file or directory
Failed to initialize automounter: No such file or directory
[FAILED] Failed to set up automount Arbitrary Executable File Forma... Automount Point.
See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details.
Unit proc-sys-fs-binfmt_misc.automount entered failed state.
[ OK ] Reached target Encrypted Volumes.
[ OK ] Listening on Journal Socket.
         Mounting Huge Pages File System...
Caught <SEGV>, dumped core as pid 11.
Freezing execution.

[Serge Hallyn (serge-hallyn)]

Please make sure that lxc.kmsg = 0 in your container configuration file for systemd guests.

[Martin Pitt (pitti)]

Retitling the bug to be more general, and adding a trusty task to reflect the original "trusty host" part. This was discussed on UOS last week (https://blueprints.launchpad.net/ubuntu/+spec/core-1411-systemd-migration). The intention is to make this work on vivid first, and then backport a newer LXC (or possibly just some patches) to trusty so that systemd containers work on trusty as well.

[Martin Pitt (pitti)]

I prepared a minimal vivid container with systemd-sysv, and tried to boot it (vivid host):

$ sudo lxc-start -n vivid-systemd -F
Failed to mount cgroup at /sys/fs/cgroup/systemd: Permission denied
[... hangs ...]

In apparmor I see:
[10072.122514] audit: type=1400 audit(1416213339.298:50): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/cgroup/systemd/" pid=16469 comm="systemd" fstype="cgroup" srcname="cgroup" flags="rw, nosuid, nodev, noexec"

After setting "lxc.aa_profile = unconfined", the container boots (with similar error message spew as in #1, which we can ignore for now), but logging in on the console takes a long time. systemd-journal (in the guest) starts spinning the CPU to 100%. "sudo journalctl" shows me the logs. stracing shows

read(9, "", 8192) = 0
epoll_wait(7, {{EPOLLIN|EPOLLERR|EPOLLHUP, {u32=3073693008, u64=140547288520016}}, {EPOLLIN, {u32=3073692768, u64=140547288519776}}, {EPOLLIN, {u32=3073692288, u64=140547288519296}}, {EPOLLIN, {u32=3073692528, u64=140547288519536}}}, 14, 0) = 4
clock_gettime(0x7 /* CLOCK_??? */, {10618, 410721720}) = 0
writev(2, [{"/dev/kmsg buffer overrun, some m"..., 45}, {"\n", 1}], 2) = 46

I tried to set "lxc.kmsg = 0" as Serge indicated in comment 2, but this doesn't seem to have the intended effect: in the container I still see "/dev/kmsg -> console".

For the record: booting and journal work fine in systemd-nspawn; but this has neither apparmor protection nor does it do the /dev/kmsg -> /dev/lxc/console trick; instead, /dev/kmsg does not exist at all there.

[Martin Pitt (pitti)]

With latest lxc and lxcfs this works now, both system and session containers with systemd, with systemd on the host.

[Christopher Townsend (townsend)]

Any ideas when this will be fixed in Trusty? This is blocking unity8-lxc from working anymore on Trusty hosts.

[Stéphane Graber (stgraber)]

Not anytime soon unfortunately. The backport of all the needed bits as SRUs will be very very tricky to get right, currently our timeframe is "by 16.04".

If you're maintaining a PPA already, you could pick up the required trusty packages from ppa:ubuntu-lxc/daily or just include your own backports of cgmanager, lxcfs and lxc. They all work great on trusty (that's what I'm using here) but we can't just push them as is into a stable release.

[gcc (chris+ubuntu-qwirx)]

Thank you Stephane for your PPA! I installed 14.04 expecting LXC to just work, found that it didn't, somehow found this page, installed your PPA and the updated utilities and it now appears to be working perfectly! I only wish it would work out of the box in an LTS release.

[Jens Elkner (jelmd)]

I'm running utopic with latest updates. Any container, which has systemd running simply hangs, when /sbin/init gets started (no matter, whether config has 'lxc.kmsg = 0' or not). Tried it previously with a trusty and today with a vivid container. So wondering, whether there is a bugfix available at all?

[Wulf Assols (bombo-q)]

I'm usingTrusty (14.04.2) trying to run debian/jessie in lxc

what i did:
$ add-apt-repository ppa:ubuntu-lxc/daily
$ apt-get update
$ apt-get install lxc cgmanager lxcfs

then:
$ lxc-create -n jessie -t debian
$ lxc-start -n jessie
nothing seems to happen

in the syslog i see:
Jul 31 20:07:05 ubu kernel: [ 962.944281] type=1400 audit(1438366025.666:88): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/cgroup/cpu,cpuacct/" pid=15251 comm="systemd" fstype="cgroup" srcname="cgroup" flags="rw, nosuid, nodev, noexec"
Jul 31 20:07:05 ubu kernel: [ 962.944321] type=1400 audit(1438366025.666:89): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="lxc-container-default" name="/sys/fs/cgroup/" pid=15251 comm="systemd" flags="ro, nosuid, nodev, noexec, remount, strictatime"

but it's actually running, i can get a shell with:
$ lxc-attach -n jessie
root@jessie:~#

so there is just the login part missing for some reason (init/systemd won't start getty?)

Thanks Stephane for the PPA!

[Stéphane Graber (stgraber)]

Fixed through trusty-backports.

[Rafał Błaszczyk (rblaszczyk)]

The problem is fixed but needs additional comment.
I've had a problem after upgrading only lxc package from trusty-backports.
The network was not working in debian jessie and after logging in (lxc-attach) systemd was not working correctly.
The solution was to update not only lxc package but also lxc-templates and recreate guest OS.

rm -rf /var/cache/lxc/debian # remove cached debian template
apt-get install -t trusty-backports lxc lxc-templates # update both lxc and lxc-templates

After that it works without a problem. Thank you for the fix.

[Jens Elkner (jelmd)]

Today I upgraded our last utopic containers (~10) to vivid using do-release-upgrade: Everywhere the same: after reboot systemd is the only thing which is running in the container, but nothing else happens. It doesn't start anything! So the only way to get the stuff fixed is to manually attach to the container, do a ' ln -sf upstart /sbin/init', logout and force a lxc-stop/start of the container and do the remaining things ...

[Serge Hallyn (serge-hallyn)]

@Jens

this is very dependent on your exact setup, so please open a new
bug specifying your precise host environment (release, any ppas,
package versions, any custom configuration, and error logs if possible)