fails to mount cgroupfs inside containers running on 16.04

[Cam Cope]

I need to run nested Ubuntu 12.04 and 14.04 containers on 16.04 hosts, and have noticed that the cgroups-mount script for mounting the cgroups inside the containers has stopped working. This is because systemd now comounts multiple controllers on a single hierarchy, which prevents mounting them individually inside the container.

=========== SRU Justification ================
Impact: nested containers fail to start
Reproduce: create a root owned container; install lxc and cgroup-lite; create a container, and try to start it. Starting will fail if cgroup-lite is running in the first level container without this patch.
Regression potential: should be low, it's possible that the regexp is simply wrong for some cases.
===============================================

[Cam Cope (ccope)]

Patch for cgroup-lite 1.1.5

[Cam Cope (ccope)]

Patch for 1.12

[Cam Cope (ccope)]

It looks like there were fixes in the latest version of cgroup-lite that would still be applicable/useful for earlier ubuntu releases, but I tried to minimize the diffs to have the functionality that I need. Let me know if you need a patch for trusty as well.

[Serge Hallyn (serge-hallyn)]

Thanks for the patches. Note that you've changed the loop variable from d to c, but the loop is still using d, so nothing gets mounted with that patch verbatim. Switching the variable back to d fixes the issue for me.

[Cam Cope (ccope)]

Whoops, didn't notice I changed that part in my local copy. Should have been more careful with my patch.

[Cam Cope (ccope)]

Patch for 1.1.5

[Cam Cope (ccope)]

Patch for 1.12

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "cgroup-lite_1.1.5.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Cam Cope (ccope)]

Also, is there a reason the name=systemd cgroup controller gets mounted at /sys/fs/cgroup/systemd instead of /sys/fs/cgroup/name=systemd in newer versions of cgroups-mount? It means my applications have to special-case stripping off the name= when they try to work with cgroups based upon /proc/cgroups.

[Cam Cope (ccope)]

Edit: re: the systemd hierarchy, it's actually not mentioned in /proc/cgroups, it's /proc/self/cgroups

[Serge Hallyn (serge-hallyn)]

I assume /sys/fs/cgroup/systemd was used because that's where systemd expected to see it. I can't find any version which mounts it at /sys/fs/cgroup/name=systemd - do you have a version that does that?

[Cam Cope (ccope)]

Hm, I found a bug in my last version of this patch. Freshly booted machines which had not mounted the cgroupfs had all the hierarchies as 0, causing all cgroups to get mounted onto a single directory. I can work around this by detecting this scenario.
However, I wonder if I am actually seeing a bug in LXC. On my 12.04 hosts spawning 12.04 containers with the nesting.conf include, the cgroupfs gets automounted inside the container even without this package. This is not the case on 16.04 hosts. I'm currently using LXC 2.0.6.

RE: name=systemd, I had modified an older version of our scripts to mount name=systemd, because that was how it showed up in /proc/self/cgroups, but everywhere I see the systemd cgroup mentioned on the internet has it mounted at /sys/fs/cgroup/systemd, so I guess that's an implementation detail I just have to deal with.

[Serge Hallyn (serge-hallyn)]

Hm, yes i see (hierarchy 0). I do worry that means that systems with the unified hierarchy will be still more of a problem, since it is mounted as hierarchy 0. But then it's probably fine to just say that cgroup-lite doesn't support unified hierarchy.

[Serge Hallyn (serge-hallyn)]

(Note - this means the patch I just pushed to zesty is in fact wrong. Since zesty requires systemd this doesn't really matter, but we must get a working patch before pushing SRU patches)

@ccope, please let me know if you care to write an update patch.

[Serge Hallyn (serge-hallyn)]

This bug incidentally also affects the cgroupfs-mount package.

[Cam Cope (ccope)]

I'm happy to provide a patch, but if the root cause of my issue is in lxc
it may be easier to patch that than worrying about backwards compatibility
for cgroups on older distro releases.

On Mar 3, 2017 20:11, "Serge Hallyn" <email address hidden> wrote:

> This bug incidentally also affects the cgroupfs-mount package.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1668724
>
> Title:
> fails to mount cgroupfs inside containers running on 16.04
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cgroup-lite/+
> bug/1668724/+subscriptions
>

[Serge Hallyn (serge-hallyn)]

There is a bug (which is fixed in newer versions) in lxc in that it does not ignore cgroups which are not used by the container. So in your particular case, if you're not using the co-mounted controllers, then indeed fixing the lxc bug should work for you as well.

Nevertheless some people will want to run containers using those controllers, so cgroup-lite (and cgroupfs-mount) will need to be fixed. But that's not your problem :) I understand if you want to leave it at this - I'll just need to improve my awk skills :) I just wanted to ask to avoid duplication of effort.

Thanks for your help so far.

[Serge Hallyn (serge-hallyn)]

FWIW something like:

for d in `tail -n +2 /tmp/2 | awk '{
        if ($2 == 0)
                print $1
        else if (a[$2])
                a[$2] = a[$2]","$1
        else
                a[$2]=$1
};END{
        for(i in a) {
                print a[i]
        }
}'`; do
        mkdir -p /sys/fs/cgroup/$d
        mountpoint -q /sys/fs/cgroup/$d || (mount -n -t cgroup -o $d cgroup /sys/fs/cgroup/$d || rmdir /sys/fs/cgroup/$d || true)
 done

seems to work. Again excluding the possibility of unified hierarchy, which I've not tested.

[Serge Hallyn (serge-hallyn)]

(s@/tmp/2@/proc/cgroups@ of course)

[Launchpad Janitor (janitor)]

This bug was fixed in the package cgroup-lite - 1.13

---------------
cgroup-lite (1.13) zesty; urgency=medium

  * Handle mounting comounted cgroup controllers (LP: #1668724)

 -- Cameron Cope <email address hidden> Tue, 28 Feb 2017 11:48:05 -0800

[Cam Cope (ccope)]

LGTM

[Serge Hallyn (serge-hallyn)]

Thanks for looking. I'll push that tonight.

  Original Message
From: Cam Cope
Sent: Monday, March 6, 2017 3:50 PM
To: <email address hidden>
Reply To: Bug 1668724
Subject: [Bug 1668724] Re: fails to mount cgroupfs inside containers running on 16.04

LGTM

--
You received this bug notification because you are subscribed to cgroup-
lite in Ubuntu.
Matching subscriptions: cgroup-lite
https://bugs.launchpad.net/bugs/1668724

Title:
fails to mount cgroupfs inside containers running on 16.04

Status in cgroup-lite package in Ubuntu:
Fix Released
Status in cgroup-lite source package in Precise:
New
Status in cgroup-lite source package in Trusty:
New
Status in cgroup-lite source package in Xenial:
New
Status in cgroup-lite source package in Yakkety:
New

Bug description:
I need to run nested Ubuntu 12.04 and 14.04 containers on 16.04 hosts,
and have noticed that the cgroups-mount script for mounting the
cgroups inside the containers has stopped working. This is because
systemd now comounts multiple controllers on a single hierarchy, which
prevents mounting them individually inside the container.

=========== SRU Justification ================
Impact: nested containers fail to start
Reproduce: create a root owned container; install lxc and cgroup-lite; create a container, and try to start it. Starting will fail if cgroup-lite is running in the first level container without this patch.
Regression potential: should be low, it's possible that the regexp is simply wrong for some cases.
===============================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cgroup-lite/+bug/1668724/+subscriptions

[Robie Basak (racb)]

Hello Cam, or anyone else affected,

Accepted cgroup-lite into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

Hello Cam, or anyone else affected,

Accepted cgroup-lite into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.11ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Robie Basak (racb)]

I see tasks for Trusty and Yakkety but they aren't in the queue?

[Robie Basak (racb)]

Unsubscribing ~ubuntu-sponsors as I don't see any debdiffs or similar here.

[Robie Basak (racb)]

Ah. I see cgroup-lite in the backports queue for Trusty. Was this intentional?

For Yakkety, I do see it in the queue manually. Probably a bug in my script. I'll investigate.

[Robie Basak (racb)]

My script can't see the Yakkety upload because it's in unapproved in the updates pocket, not the proposed pocket. I've never seen that before. How did you upload it?

[Serge Hallyn (serge-hallyn)]

Sorry, this got lost in my inbox. I used dput. I seemed to recall that in some recent cycle the it became either ok or preferred to use $release-updates instead of $release-proposed in SRUs. I can trivially change it ack to yakkety-proposed if preferred.

For Trusty, I just noticed that the non-backports version was so old noone could really be using it (and AIUI trusty-backport is enabled by default).

[Chris J Arges (arges)]

Hello Cam, or anyone else affected,

Accepted cgroup-lite into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.11ubuntu0.16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Ubuntu Foundations Team Bug Bot (crichton)]

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

[Łukasz Zemczak (sil2100)]

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

[Andy Whitcroft (apw)]

The version of cgroup-lite in the proposed pocket of Precise that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

[Christian Ehrhardt  (paelzer)]

This was removed from Xenial-Proposed as well.
It is still in Y-proposed but that is EOL.

Given the time that has passed and the rather unclear state I'd consider this bug incomplete.
In comment #29 Serge outlined that for Trusty all of that not really applies (everyone is using backports there?)

I'll update the bug tasks to reflect all that.

@Serge - Please update if you still work on this or have a fix somewhere.

[Serge Hallyn (serge-hallyn)]

Drat. I do think this should still be pushed. I don't know when I'll have time to do it though. Please keep it open.

[Serge Hallyn (serge-hallyn)]

I'll still aim to push this for trusty and xenial.