xenial systemd fails to start if unified-only (non-hybrid) cgroup2 is mounted on jammy hosts

[Dan Streetman]

[impact]

now that jammy has moved to using unified cgroup2, containers started on jammy must also use unified cgroup2 (since the cgroup subsystems can only be mounted as v1 or v2 throughout the entire system, including inside containers).

However, the systemd in xenial does not include support for cgroup2, and doesn't recognize its magic (added in upstream commit 099619957a0), so it fails to start completely.

[workaround]
On Jammy host edit default kernel command line to include

systemd.unified_cgroup_hierarchy=false

update your bootloader configuration; and reboot

then hybrid cgroups will be on the host, and one can launch xenial container then.

[test case]

create a jammy system, that has unified cgroup2 mounted. Then:

$ lxc launch ubuntu:xenial test-x
...
$ lxc shell test-x

(inside xenial container):
$ mv /sbin/init /sbin/init.old
$ cat > /sbin/init <<EOF
#!/bin/bash
sleep 2
exec /lib/systemd/systemd --log-level=debug --log-target=console
EOF
$ chmod 755 /sbin/init
$ exit

(back in jammy host system):
$ lxc stop test-x -f
$ lxc start --console test-x
To detach from the console, press: <ctrl>+a q
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems, freezing.
Freezing execution.

[regression potential]

any regression would likely break xenial containers from starting at all, or cause cgroup-related problems with systemd starting and/or managing services.

[scope]

this is needed only for xenial. However, as xenial is out of standard support, this would need to be an exception.

this is fixed upstream with commit 099619957a0 (and possibly others - needs closer investigation and testing) which is first included in v230, so this is fixed already in b and later. However that patch alone is insufficient, does not implement hybrid group setup, and breaks a lot of xenial userspace expectations.

this is not needed - by default - for trusty because upstart is used there; however, I think it's possible to change trusty over to use systemd instead of upstart. But since trusty is out of standard support, and it doesn't fail by default, it doesn't seem like it should be fixed.

[other info]

An alternative appears to be to change the host system back to using the 'hybrid' cgroup, however that obviously is awful and would remove the benefits of cgroup v2 from the host system, and force all containers on the host system to also use the 'hybrid' cgroup.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Frank Heimes (fheimes)]

This bug is a consequence of LP#1962286.

[Michael Hudson-Doyle (mwhudson)]

"so this is fixed already in f and later" - think you mean "b and later" here?

[Steve Langasek (vorlon)]

NB this has been assigned to Canonical Foundations, but as Ubuntu 16.04 is in Extended Security Maintenance now, this is actually a decision for our ESM Team to make regarding the path forward on whether xenial containers will be supported on jammy hosts, and if so, to update systemd for it.

[Steve Langasek (vorlon)]

(not reassigning because I'm not sure of a public team that can be used for ESM bug assignments, but I've contacted the engineering team internally.)

[Dan Streetman (ddstreet)]

> "so this is fixed already in f and later" - think you mean "b and later" here?

yes sorry, fixed in b and later

[Dimitri John Ledkov (xnox)]

Irrespective of ESM status, we have always had extremely long support overlaps both backwards and forwards between ubuntu releases.

At the moment, my only solution is to use lxd vms; i.e.
do

$ lxc launch --vm ubuntu:xenial

However, I say for the sake of ease of development, testing, upgrades, migration, and bug hunting we should support xenial lxd on jammy, irrespective of xenial's status, especially since trusty lxd on jammy still works.

[Dimitri John Ledkov (xnox)]

hm, $ lxc launch --vm ubuntu:xenial fails for me

[Dimitri John Ledkov (xnox)]

Backporting 099619957a0 to xenial will mean that systemd will gain ability to use cgroups2 as shipped in the xenial's ga v4.4 kernel.

it will mean that xenial containers on top of bionic's ga kernel will fail to use cgroups2.

however at the time it was an experimental feature which was not widely used at all, and there are likely to be very few users of it.

it would be nice to if backport of 099619957a0 was done in a backwards-compatible way and allow using cgroups2 like code paths using both pre-v4.4 and v4.4+ kernels.

[Leonidas S. Barbosa (leosilvab)]

I have update ongoing on esm-infa-staging for it, but without consider these issues Dimitri pointed, so, with the clean patch pointed here. It would be nice if any of you folks could give a test on it. Also adding other folks from security to this discussion with better background.

Also, tips on how to test it would be nice. I was trying lxc image export with a systemd updated and a non systemd update xenial, but hadn't any luck as both work as the same if you lxc import into a jammy from .tar.gz image exported.

[Dan Streetman (ddstreet)]

> I have update ongoing on esm-infa-staging for it

just to clarify, if the systemd patch isn't added to the actual main (currently closed) xenial-updates repo, it won't be very much help to anyone wanting to create new xenial containers on jammy. So, just patching systemd in the esm repo likely is not enough - especially if the prebuilt LXD container images don't pick up the ESM version of systemd.

[Frank Heimes (fheimes)]

+1 to Dan
Having it in ESM is not really helpful (esp. not for package tests).

If it's patched in the regular archives (xenial-updates), I am happy to re-try and test.

[Leonidas S. Barbosa (leosilvab)]

I see, I do agree, sorry for noise. I did remove the one from -esm ppas and here is the debdiff. Could someone please sponsor/follow with it? Thanks!

[Leonidas S. Barbosa (leosilvab)]

Just for clarification, and doc, this approach was took as since it needs to go the archive, it need to be handled as an SRU bye the SRU team.

[Lukas Märdian (slyon)]

I feel like this special case should be decided and handled by the release team, not Foundations.

[Steve Langasek (vorlon)]

It's the SRU team's purview to make exceptions here. The rationale that this only benefits images is and therefore needs to be in the main archive is sound. So for the SRU team, I'm +1 on going ahead with this. But someone still needs to prepare the SRU, which I think falls to Foundations.

[Dimitri John Ledkov (xnox)]

I've attempted to prepare naive SRUs and it doesn't get me far yet. The issue is that unified cgroups2 support in xenial's systemd is rudimental, and falls apart very quickly with none of the xenial userspace expecting or able to work correctly with unified cgroups2 setup.

I fear that it will not be practical to SRU systemd into xenial to resolve this. And users on Jammy that need to launch xenial containers should continue to switch their systems to hybrid cgroups setup by using `systemd.unified_cgroup_hierarchy=false` kernel command line option on their hosts.

[Dimitri John Ledkov (xnox)]

From lennart:

"""
What's stopping you from mounting a private "named" cgroup v1
hierarchy to such containers (i.e. no controllers). systemd will then
use that when taking over and not bother with mounting anything on its
own, such as a cgroupv2 tree.

that should be enough to make old systemd happy.

"""

We need to try to do that, i.e. potentially tweaking lxd metadata tarball to do such a mount ahead of starting a container.