fails to mount cgroupfs inside containers running on 16.04

Bug #1668724 reported by Cam Cope
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
cgroup-lite (Ubuntu)
Fix Released
High
Unassigned
Precise
Won't Fix
High
Unassigned
Trusty
Incomplete
High
Unassigned
Xenial
Incomplete
High
Unassigned
Yakkety
Won't Fix
High
Unassigned

Bug Description

I need to run nested Ubuntu 12.04 and 14.04 containers on 16.04 hosts, and have noticed that the cgroups-mount script for mounting the cgroups inside the containers has stopped working. This is because systemd now comounts multiple controllers on a single hierarchy, which prevents mounting them individually inside the container.

=========== SRU Justification ================
Impact: nested containers fail to start
Reproduce: create a root owned container; install lxc and cgroup-lite; create a container, and try to start it. Starting will fail if cgroup-lite is running in the first level container without this patch.
Regression potential: should be low, it's possible that the regexp is simply wrong for some cases.
===============================================

Revision history for this message
Cam Cope (ccope) wrote :

Patch for cgroup-lite 1.1.5

Revision history for this message
Cam Cope (ccope) wrote :

Patch for 1.12

Changed in cgroup-lite (Ubuntu):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Cam Cope (ccope) wrote :

It looks like there were fixes in the latest version of cgroup-lite that would still be applicable/useful for earlier ubuntu releases, but I tried to minimize the diffs to have the functionality that I need. Let me know if you need a patch for trusty as well.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for the patches. Note that you've changed the loop variable from d to c, but the loop is still using d, so nothing gets mounted with that patch verbatim. Switching the variable back to d fixes the issue for me.

Revision history for this message
Cam Cope (ccope) wrote :

Whoops, didn't notice I changed that part in my local copy. Should have been more careful with my patch.

Revision history for this message
Cam Cope (ccope) wrote :

Patch for 1.1.5

Revision history for this message
Cam Cope (ccope) wrote :

Patch for 1.12

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "cgroup-lite_1.1.5.patch" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Cam Cope (ccope) wrote :

Also, is there a reason the name=systemd cgroup controller gets mounted at /sys/fs/cgroup/systemd instead of /sys/fs/cgroup/name=systemd in newer versions of cgroups-mount? It means my applications have to special-case stripping off the name= when they try to work with cgroups based upon /proc/cgroups.

Revision history for this message
Cam Cope (ccope) wrote :

Edit: re: the systemd hierarchy, it's actually not mentioned in /proc/cgroups, it's /proc/self/cgroups

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I assume /sys/fs/cgroup/systemd was used because that's where systemd expected to see it. I can't find any version which mounts it at /sys/fs/cgroup/name=systemd - do you have a version that does that?

Changed in cgroup-lite (Ubuntu Precise):
importance: Undecided → High
Changed in cgroup-lite (Ubuntu Trusty):
importance: Undecided → High
Changed in cgroup-lite (Ubuntu Xenial):
importance: Undecided → High
Changed in cgroup-lite (Ubuntu Yakkety):
importance: Undecided → High
description: updated
Revision history for this message
Cam Cope (ccope) wrote :

Hm, I found a bug in my last version of this patch. Freshly booted machines which had not mounted the cgroupfs had all the hierarchies as 0, causing all cgroups to get mounted onto a single directory. I can work around this by detecting this scenario.
However, I wonder if I am actually seeing a bug in LXC. On my 12.04 hosts spawning 12.04 containers with the nesting.conf include, the cgroupfs gets automounted inside the container even without this package. This is not the case on 16.04 hosts. I'm currently using LXC 2.0.6.

RE: name=systemd, I had modified an older version of our scripts to mount name=systemd, because that was how it showed up in /proc/self/cgroups, but everywhere I see the systemd cgroup mentioned on the internet has it mounted at /sys/fs/cgroup/systemd, so I guess that's an implementation detail I just have to deal with.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Hm, yes i see (hierarchy 0). I do worry that means that systems with the unified hierarchy will be still more of a problem, since it is mounted as hierarchy 0. But then it's probably fine to just say that cgroup-lite doesn't support unified hierarchy.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

(Note - this means the patch I just pushed to zesty is in fact wrong. Since zesty requires systemd this doesn't really matter, but we must get a working patch before pushing SRU patches)

@ccope, please let me know if you care to write an update patch.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

This bug incidentally also affects the cgroupfs-mount package.

Revision history for this message
Cam Cope (ccope) wrote : Re: [Bug 1668724] Re: fails to mount cgroupfs inside containers running on 16.04

I'm happy to provide a patch, but if the root cause of my issue is in lxc
it may be easier to patch that than worrying about backwards compatibility
for cgroups on older distro releases.

On Mar 3, 2017 20:11, "Serge Hallyn" <email address hidden> wrote:

> This bug incidentally also affects the cgroupfs-mount package.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1668724
>
> Title:
> fails to mount cgroupfs inside containers running on 16.04
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/cgroup-lite/+
> bug/1668724/+subscriptions
>

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

There is a bug (which is fixed in newer versions) in lxc in that it does not ignore cgroups which are not used by the container. So in your particular case, if you're not using the co-mounted controllers, then indeed fixing the lxc bug should work for you as well.

Nevertheless some people will want to run containers using those controllers, so cgroup-lite (and cgroupfs-mount) will need to be fixed. But that's not your problem :) I understand if you want to leave it at this - I'll just need to improve my awk skills :) I just wanted to ask to avoid duplication of effort.

Thanks for your help so far.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

FWIW something like:

for d in `tail -n +2 /tmp/2 | awk '{
        if ($2 == 0)
                print $1
        else if (a[$2])
                a[$2] = a[$2]","$1
        else
                a[$2]=$1
};END{
        for(i in a) {
                print a[i]
        }
}'`; do
        mkdir -p /sys/fs/cgroup/$d
        mountpoint -q /sys/fs/cgroup/$d || (mount -n -t cgroup -o $d cgroup /sys/fs/cgroup/$d || rmdir /sys/fs/cgroup/$d || true)
 done

seems to work. Again excluding the possibility of unified hierarchy, which I've not tested.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

(s@/tmp/2@/proc/cgroups@ of course)

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package cgroup-lite - 1.13

---------------
cgroup-lite (1.13) zesty; urgency=medium

  * Handle mounting comounted cgroup controllers (LP: #1668724)

 -- Cameron Cope <email address hidden> Tue, 28 Feb 2017 11:48:05 -0800

Changed in cgroup-lite (Ubuntu):
status: Triaged → Fix Released
Revision history for this message
Cam Cope (ccope) wrote :

LGTM

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Thanks for looking. I'll push that tonight.

  Original Message
From: Cam Cope
Sent: Monday, March 6, 2017 3:50 PM
To: <email address hidden>
Reply To: Bug 1668724
Subject: [Bug 1668724] Re: fails to mount cgroupfs inside containers running on 16.04

LGTM

--
You received this bug notification because you are subscribed to cgroup-
lite in Ubuntu.
Matching subscriptions: cgroup-lite
https://bugs.launchpad.net/bugs/1668724

Title:
fails to mount cgroupfs inside containers running on 16.04

Status in cgroup-lite package in Ubuntu:
Fix Released
Status in cgroup-lite source package in Precise:
New
Status in cgroup-lite source package in Trusty:
New
Status in cgroup-lite source package in Xenial:
New
Status in cgroup-lite source package in Yakkety:
New

Bug description:
I need to run nested Ubuntu 12.04 and 14.04 containers on 16.04 hosts,
and have noticed that the cgroups-mount script for mounting the
cgroups inside the containers has stopped working. This is because
systemd now comounts multiple controllers on a single hierarchy, which
prevents mounting them individually inside the container.

=========== SRU Justification ================
Impact: nested containers fail to start
Reproduce: create a root owned container; install lxc and cgroup-lite; create a container, and try to start it. Starting will fail if cgroup-lite is running in the first level container without this patch.
Regression potential: should be low, it's possible that the regexp is simply wrong for some cases.
===============================================

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/cgroup-lite/+bug/1668724/+subscriptions

Revision history for this message
Robie Basak (racb) wrote : Please test proposed package

Hello Cam, or anyone else affected,

Accepted cgroup-lite into precise-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.1.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cgroup-lite (Ubuntu Precise):
status: New → Fix Committed
tags: added: verification-needed
Changed in cgroup-lite (Ubuntu Xenial):
status: New → Fix Committed
Revision history for this message
Robie Basak (racb) wrote :

Hello Cam, or anyone else affected,

Accepted cgroup-lite into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.11ubuntu0.16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Revision history for this message
Robie Basak (racb) wrote :

I see tasks for Trusty and Yakkety but they aren't in the queue?

Revision history for this message
Robie Basak (racb) wrote :

Unsubscribing ~ubuntu-sponsors as I don't see any debdiffs or similar here.

Revision history for this message
Robie Basak (racb) wrote :

Ah. I see cgroup-lite in the backports queue for Trusty. Was this intentional?

For Yakkety, I do see it in the queue manually. Probably a bug in my script. I'll investigate.

Revision history for this message
Robie Basak (racb) wrote :

My script can't see the Yakkety upload because it's in unapproved in the updates pocket, not the proposed pocket. I've never seen that before. How did you upload it?

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Sorry, this got lost in my inbox. I used dput. I seemed to recall that in some recent cycle the it became either ok or preferred to use $release-updates instead of $release-proposed in SRUs. I can trivially change it ack to yakkety-proposed if preferred.

For Trusty, I just noticed that the non-backports version was so old noone could really be using it (and AIUI trusty-backport is enabled by default).

Revision history for this message
Chris J Arges (arges) wrote :

Hello Cam, or anyone else affected,

Accepted cgroup-lite into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cgroup-lite/1.11ubuntu0.16.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in cgroup-lite (Ubuntu Yakkety):
status: New → Fix Committed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [cgroup-lite/precise] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for precise for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Change of SRU verification policy

As part of a recent change in the Stable Release Update verification policy we would like to inform that for a bug to be considered verified for a given release a verification-done-$RELEASE tag needs to be added to the bug where $RELEASE is the name of the series the package that was tested (e.g. verification-done-xenial). Please note that the global 'verification-done' tag can no longer be used for this purpose.

Thank you!

Revision history for this message
Andy Whitcroft (apw) wrote : Proposed package removed from archive

The version of cgroup-lite in the proposed pocket of Precise that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in cgroup-lite (Ubuntu Precise):
status: Fix Committed → Won't Fix
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

This was removed from Xenial-Proposed as well.
It is still in Y-proposed but that is EOL.

Given the time that has passed and the rather unclear state I'd consider this bug incomplete.
In comment #29 Serge outlined that for Trusty all of that not really applies (everyone is using backports there?)

I'll update the bug tasks to reflect all that.

@Serge - Please update if you still work on this or have a fix somewhere.

Changed in cgroup-lite (Ubuntu Yakkety):
status: Fix Committed → Won't Fix
Changed in cgroup-lite (Ubuntu Xenial):
status: Fix Committed → Incomplete
Changed in cgroup-lite (Ubuntu Trusty):
status: New → Incomplete
Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

Drat. I do think this should still be pushed. I don't know when I'll have time to do it though. Please keep it open.

Revision history for this message
Serge Hallyn (serge-hallyn) wrote :

I'll still aim to push this for trusty and xenial.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.