lxc cgroup2: containers unbootable

[Jens Elkner]

Distributor ID: Ubuntu
Description: Ubuntu 20.04.2 LTS
Release: 20.04
Codename: focal

If one sets in /etc/default/grub (as e.g. desired by facebook oomd):
GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1 swapaccount=1 ..."
lxc is not able to start any containers anymore.

# lxc-start -F n04-01
lxc-start: n04-01: conf.c: lxc_setup_boot_id: 3249 Permission denied - Failed to mount /dev/.lxc-boot-id to /proc/sys/kernel/random/boot_id
                                                           Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...

config:
-------
# Common configuration
lxc.include = /usr/share/lxc/config/ubuntu.common.conf

# Container specific configuration
lxc.apparmor.profile = lxc-default-cgns-with-mounting
lxc.start.auto = 1
lxc.rootfs.path = dir:/zones/n04-01/rootfs
lxc.rootfs.options = noatime
lxc.mount.fstab = /zones/n04-01/fstab
lxc.uts.name = n04-01
lxc.arch = amd64

# Network configuration
lxc.net.0.type = macvlan
lxc.net.0.macvlan.mode = bridge
lxc.net.0.flags = up
lxc.net.0.link = vlan2
lxc.net.0.hwaddr = 00:80:41:22:0d:10
lxc.net.0.name = n04-01_0

#lxc.include = /zones/n04-01/nvconfig

lxc.start.order = 16
#lxc.net.0.ipv4 = 10.2.1.65/16
#lxc.net.0.ipv4.gateway = 10.2.0.1

[Stéphane Graber (stgraber)]

The error you're getting is coming from systemd in the container which for some reason is trying to mount a cgroup1 hierarchy rather than using cgroup2 like the rest of your system.

You may be able to workaround that by using `lxc.init_cmd` to pass /sbin/init with additional arguments to change the systemd behavior in the container.

[Jens Elkner (jelmd)]

Hmmm, that's an old one. IIRC the real root cause was, that some files possibly included via /usr/share/lxc/config/ubuntu.common.conf still used lxc.cgroup.devices.* instead of lxc.cgroup2.devices.* (the machine was upgraded from bionic to focal).

[Serge Hallyn (serge-hallyn)]

FWIW this is affecting me on jammy too. I'll have to take a look at systemd sources. Adding:

lxc.init.cmd = /sbin/init systemd.unified_cgroup_hierarchy

to my config does not help, nor does bind mounting a /proc/filesystems without 'cgroup' (v1) in it.