Switch to "unified" cgroup hierarchy (cgroupv2)

[Balint Reczey]

Systemd upstream switched the default cgroup hierarchy to unified with v243. This change is reverted by the Ubuntu systemd packages, but as unified is the way to go per upstream support should be added to all relevant Ubuntu packges (and snaps):

https://github.com/systemd/systemd/blob/v243/NEWS#L56

        * systemd now defaults to the "unified" cgroup hierarchy setup during
          build-time, i.e. -Ddefault-hierarchy=unified is now the build-time
          default. Previously, -Ddefault-hierarchy=hybrid was the default. This
          change reflects the fact that cgroupsv2 support has matured
          substantially in both systemd and in the kernel, and is clearly the
          way forward. Downstream production distributions might want to
          continue to use -Ddefault-hierarchy=hybrid (or even =legacy) for
          their builds as unfortunately the popular container managers have not
          caught up with the kernel API changes.

Systemd is rebuilt using the new default and is available from the following PPA for testing:

https://launchpad.net/~rbalint/+archive/ubuntu/systemd-unified-cgh

The autopkgtest results against other packges are available here:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan-rbalint-systemd-unified-cgh/?format=plain

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-focal-rbalint-systemd-unified-cgh/?format=plain

lxc autopkgtest failing:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan-rbalint-systemd-unified-cgh/eoan/amd64/d/docker.io/20191030_155944_2331e@/log.gz

snapd autopkgtest failing:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan-rbalint-systemd-unified-cgh/eoan/amd64/s/snapd/20191030_161354_94b26@/log.gz

docker.io autopkgtest failing:

https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-eoan-rbalint-systemd-unified-cgh/eoan/amd64/d/docker.io/20191030_155944_2331e@/log.gz

[Maciej Borzecki (maciek-borzecki)]

There's some ongoing work in snapd in this area. With 2.42 the snaps do not outright fail and a warning is printed out for the user. The current work on a named snapd v1 hierarchy should restore the snap process tracking capabilities.

[Ryutaroh Matsumoto (emojifreak)]

When Ubuntu Eoan is started with systemd.unified_cgroup_hierarchy, lxc-start (version 3.0.4 packaged by Eoan) cannot be used in its default setting. It is a combination of unsuitable default configuration and an upstream bug in LXC 3.0.4. For detail, please refer to https://github.com/lxc/lxc/issues/3183

[Ryutaroh Matsumoto (emojifreak)]

This was reported to the upstream https://github.com/lxc/lxc/issues/3198
The purpose of libpam-cgfs is only chowning some CGroup directories to the login user.
When Linux is booted with systemd.unified_cgroup_hierarchy,
/sys/fs/cgroup/user.slice/user-$UID.slice/session-nnn.scope is not chowned to a login user.
So libpam-cgfs completely fails to function under cgroup v2.

[Christian Brauner (cbrauner)]

https://github.com/lxc/lxc/issues/3198#issuecomment-562064091

[Ryutaroh Matsumoto (emojifreak)]

https://github.com/lxc/lxc/issues/3198#issuecomment-562252884

[Ryutaroh Matsumoto (emojifreak)]

https://github.com/lxc/lxc/issues/3221 Another LXC-container-doesn't-start-at-all type issue also observed on Ubuntu Eoan with systemd.unified_cgroup_hierarchy as well as Fedora 31.

[Christian Brauner (cbrauner)]

On Mon, Dec 09, 2019 at 08:41:18PM -0000, Ryutaroh Matsumoto wrote:
> https://github.com/lxc/lxc/issues/3221 Another LXC-container-doesn't
> -start-at-all type issue also observed on Ubuntu Eoan with
> systemd.unified_cgroup_hierarchy as well as Fedora 31.

That seems specific to LXC stable-3.0 which had barebone unified
hierarchy support to deal with systemd hyrbid cgroup layouts. However
the changes to git master which enable full cgroup2 compatibility have
been backported to the stable-3.0 branch and will be released with the
next bugfix release. In other words, the start-at-all on a pure unified
layout with 3.0.4 is expected unfortunately.

[Ryutaroh Matsumoto (emojifreak)]

https://github.com/lxc/lxd/issues/6587 When Ubuntu Eoan is booted with systemd.unified_cgroup_hierarchy, LXD cannot run Ubuntu Eoan in its container, but a small change to lxd/lxd/container_lxc.go enables LXD to operate as usual.

[Ryutaroh Matsumoto (emojifreak)]

lxc-checkpoint in the latest github master branch does not work under pure cgroup v2 as https://github.com/lxc/lxc/issues/3240

[Stéphane Graber (stgraber)]

LXD, LXCFS and LXC all have cgroupv2 support now.
It's certainly not perfect and things like CRIU (lxc-checkpoint) will not work until such time as cgroupv2 support is fully on part in the kernel with cgroupv1 and the needed additional interfaces are added to projects like CRIU.

But for normal day to day use, we should be in pretty good shape now.

[Balint Reczey (rbalint)]

Thank you everyone for implementing cgroup v2 support

Snapd is not reported here to be fixed, but it may be:
https://github.com/ubports/ubports-installer/issues/1448

@maciek-borzecki could you please confirm that snapd is fixed?

Debian plans switching systemd to use cgroupv2 by default and if every package listed as affected here is ready I plan making the switch in Ubuntu, too.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943981

[Michael Vogt (mvo)]

@rbalint Thanks for this heads up. Unfortunately we are not ready for cgroups v2. Snapd is working on v2 systems but a lot of the functionality is not ported. AIUI it requires quite a bit of work on our side and the two are quite different :/

[Michael Vogt (mvo)]

@rbalint Do you have a timeline when you plan this? The changes required make this most likely something we can only tackle during the 21.10 cycle :/

[Balint Reczey (rbalint)]

@mvo:

Fedora switched in 2019: https://medium.com/nttlabs/cgroup-v2-596d035be4d7
Debian switched with systemd 247.2-2 https://tracker.debian.org/news/1204112/accepted-systemd-2472-2-source-into-unstable/

I was about to follow Debian in systemd, but I'm holding the switch back for now. Could you please provide a link where snapd's progress can be tracked?

I plan keeping the current systemd default then for 21.04 to minimize disruption and give some more time for preparation, but I'd like to make the switch early in the 21.10 cycle to also have time to fix regressions by 21.10's release.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Michael Vogt (mvo)]

After discussing this we decided that we will leave cgroups v1 support for 21.04 because the snapd team will not be able to port all features to v2 in time. But early in the 21.10 cycle v1 is turned off and snapd needs to be ported to full v2 support.

[Mathew Hodson (mhodson)]

lxc (1:4.0.4-0ubuntu1) groovy; urgency=medium

  * New upstream bugfix release (4.0.4):
    - Support for new Linux clone flags (clone into cgroup)
    - Support for new Linux VFS system calls
    - Internal symbols are now properly hidden from external consumers
  * New upstream bugfix release (4.0.3):
    - Improvement to cgroupv1/cgroupv2 handling
    - Various improvements and tests for lxc-usernsexec

 -- Stéphane Graber <email address hidden> Thu, 20 Aug 2020 18:07:53 -0400

[Balint Reczey (rbalint)]

@mvo Please keep this bug updated and/or notify me when snapd is ready for the switch. Ideally way before 21.10 Feature Freeze.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Balint Reczey (rbalint)]

Docker supports cgroups v2 starting with version 20.10: https://www.docker.com/blog/introducing-docker-engine-20-10/

[Lukas Märdian (slyon)]

After enabling the unified cgroups hierarchy in systemd and running some tests, I can confirm that most things are working as expected, except for snapd.

https://bileto.ubuntu.com/excuses/4602/impish.html

The Bileto autopkgtests show (ignoring two unrelated/flaky failures in universe packages: suricata & mosquitto) that:

* The systemd "tests-in-lxd" test fails, for snapd's "WARNING: cgroup v2 is not fully supported yet, proceeding with partial confinement" stderr output. If that stderr output is muted/ignored (by modifying autopkgtests itself) the test passes, as tested locally.

* The snapd "autopkgtest:adt-local:tests/smoke/sandbox" spread test fails with an "all ubuntu systems must have strict confinement" error message.

[Lukas Märdian (slyon)]

snapd landed the support in upstream git as of yesterday (https://github.com/snapcore/snapd) and it is expected to be shipped with snapd v2.53

[Launchpad Janitor (janitor)]

This bug was fixed in the package systemd - 248.3-1ubuntu7

---------------
systemd (248.3-1ubuntu7) impish; urgency=medium

  * d/tests/tests-in-lxd: suppress the cgroups v2 warning on stderr from
    lxd/lxc even more comprehensively until the snapd change required to
    do it nicely gets into a release.

 -- Michael Hudson-Doyle <email address hidden> Wed, 01 Sep 2021 19:21:58 +1200

[Launchpad Janitor (janitor)]

This bug was fixed in the package snapd - 2.53~pre1.git19b68f708

---------------
snapd (2.53~pre1.git19b68f708) impish; urgency=medium

  * New git snapshot with fix for snapd.seeded.service in lxd
    (LP: #1944004)
  * New git snapshot that supports cgroups v2 (LP: #1850667)

 -- Michael Vogt <email address hidden> Fri, 24 Sep 2021 16:26:55 +0200