cgroup-lite and separated /usr

[Norberto Bensa]

============================================
SRU Justification:
1. Impact: cgroup-lite fails to start on systems with separate /usr
2. Development fix: move cgroup-{,u}mount commands to /bin
3. Stable fix: same as development fix
4. Test case:
   1. install cgroup-lite and lxc
   2. start a shell with /usr over-mounted
      2a. sudo lxc-unshare -s MOUNT /bin/bash
      2b. mount -t tmpfs tmpfs /usr
      2c. /bin/cgroup-umount # look for errors
      2d. look under /sys/fs/cgroup - cgroups should be mounted
      2e. /bin/cgroup-mount # look for errors
      2f. look under /sys/fs/cgroup - cgroups should be mounted
      # (note, if you do 'start cgroup-lite from the private mounts namespace, the mounts will only be seen in the host's (upstart's) namespace, not your private namespace)
5. Regression potential: If done wrong, it could cause cgroup-lite to fail
   to start or stop.
============================================

the initscript provided by cgroup-lite 1.1 doesn't work when /usr is on separated partition.

$ sudo lsb_release -rd
Description: Ubuntu 12.04 LTS
Release: 12.04

$ apt-cache policy cgroup-lite
cgroup-lite:
  Installed: 1.1
  Candidate: 1.1
  Version table:
 *** 1.1 0
        500 http://archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

with separated /usr partition:
$ mount | grep /usr
/dev/mapper/venkman-usr on /usr type ext4 (rw,noatime)
$ sudo service cgroup-lite status
cgroup-lite stop/waiting
$ mount | grep cgroup

with not separated /usr partition:
$ mount | grep /usr
$ sudo service cgroup-lite status
cgroup-lite start/running
$ mount | grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Andreas Hasenack (ahasenack)]

Same here, I have /usr separated and cgroup isn't mounted after boot. This breaks LXC and juju local deployments.

When I run it manually after booting (/etc/init.d/cgroup-lite I mean), I get only one mount point, lxc seems to work:

root@nsn7:~# mount|grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)

[akrus (akrus-flygroup)]

Confirming the problem, cgroup is not mounted automatically, thus failing to autostart the containers...
As a workaround I may suggest installing cgroup-bin instead of cgroup-lite.

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug. I'll move the scripts to /bin, with symbolic links in /usr/bin/

[Launchpad Janitor (janitor)]

This bug was fixed in the package cgroup-lite - 1.2

---------------
cgroup-lite (1.2) quantal; urgency=low

  * move cgroups-mount and cgroups-umount to /bin, with links in /usr
    (LP: #989354)
  * cgroups-mount: log and exit (with success) if any cgroup fs is mounted
    anywhere (LP: #985884)
  * cgroups-umount: be robust if /sys/fs/cgroup does not exist.
 -- Serge Hallyn <email address hidden> Wed, 20 Jun 2012 12:57:33 -0500

[Brian Murray (brian-murray)]

Hello zoolook, or anyone else affected,

Accepted cgroup-lite into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cgroup-lite/1.1.1 in a few hours and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case details of your testing will help us make a better decision. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Norberto Bensa (nbensa)]

Hello,

it is better than before, but not fixed yet (at least for me)

$ mount|grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)

$ grep cgroup /proc/mounts
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0

$ apt-cache policy cgroup-lite
cgroup-lite:
  Instalados: 1.1.1
  Candidato: 1.1.1
  Tabla de versión:
 *** 1.1.1 0
        500 http://archive.ubuntu.com/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1 0
        500 http://ar.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

$ mount|grep /usr
/dev/mapper/vg_lxc-usr on /usr type ext4 (rw)

$ cat /proc/cgroups
#subsys_name hierarchy num_cgroups enabled
cpuset 0 1 1
cpu 0 1 1
cpuacct 0 1 1
memory 0 1 1
devices 0 1 1
freezer 0 1 1
blkio 0 1 1
perf_event 0 1 1

$ grep cgroup /etc/fstab
$

Thanks!

[Serge Hallyn (serge-hallyn)]

Sorry about that. I'll try to reproduce.

[Norberto Bensa (nbensa)]

Hello Serge,

don't worry.

Does it make any difference to test on a vm (virtualbox on mac os x)?

Also, if I stop/start cgroup-lite, lxc guest runs

Fresh boot:

$ sudo lxc-start -n test
[sudo] password for zoolook:
lxc-start: failed to spawn 'test'

$ sudo service cgroup-lite stop
cgroup-lite stop/waiting

$ sudo service cgroup-lite start
cgroup-lite start/running

$ cat /proc/mounts |grep cgroup
cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset,clone_children 0 0
cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu,clone_children 0 0
cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct,clone_children 0 0
cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory,clone_children 0 0
cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices,clone_children 0 0
cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer,clone_children 0 0
cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio,clone_children 0 0
cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event,clone_children 0 0

$ mount|grep cgroup
cgroup on /sys/fs/cgroup type tmpfs (rw,uid=0,gid=0,mode=0755)

(why doesn't show all of them?)

$ sudo lxc-start -n test
(now it doesn't complain, and the container runs)

I hope this helps

Regards,
Norberto

[Serge Hallyn (serge-hallyn)]

A vm versus a real machine should make no difference.

> (why doesn't show all of them?)

Mount looks at /etc/mtab, which in Ubuntu is a separate file,
not linked to /proc/mounts. The cgroup filesystems are mounted
with '-n' to not add entries to mtab.

> $ sudo lxc-start -n test
> (now it doesn't complain, and the container runs)

It shows that there is something missing from /usr. Actually
I'm wondering whether /proc might be not yet available when
/sys is available, as that would explain it, but I wouldn't
think that would be done.

I just need to set up a new test machine and see what's going
on.

[Serge Hallyn (serge-hallyn)]

Hm, I can't reproduce this in a VM with separate /usr, with precise-proposed in my sources.list.

Can you change the 'start on' in /etc/init/cgroup-lite.conf to be:

   start on mounted MOUNTPOINT=/sys and mounted MOUNTPOINT=/proc

If that still fails, then we'll need to add some debugging output to your job.

[Norberto Bensa (nbensa)]

Hello Serge,

[quote]
start on mounted MOUNTPOINT=/sys and mounted MOUNTPOINT=/proc
[/quote]

A few months ago, I got a similar problem at work. I needed to make sure two filesystems were mounted, and I came to a solution like the one you're suggesting. And nope. It didn't work. cgroup-lite is in an endless loop right now :-)

So I tried to put some echoes on the job. And the problem is.... (are actually) tail and awk

awk is installed in /usr/bin/awk (which is a symlink to /etc/alternatives/akw, which is a symlink to /usr/bin/mawk). The same for tail.

The for loop (for c in `tail -n +2 /proc/cgroups | awk....) at the end of /bin/cgroup-mount is never executed.

I did a simple "cp /usr/bin/awk /bin/" and "cp /usr/bin/tail /bin/" and got cgroup-lite working.

Now I'm wondering two things:

1) how did you get this to work with a separated /usr

2) what others things are broken when /usr is on a separated partition.

Regards,
Norberto

[Serge Hallyn (serge-hallyn)]

Quoting zoolook (<email address hidden>):
> Hello Serge,
>
> [quote]
> start on mounted MOUNTPOINT=/sys and mounted MOUNTPOINT=/proc
> [/quote]
>
> A few months ago, I got a similar problem at work. I needed to make sure
> two filesystems were mounted, and I came to a solution like the one
> you're suggesting. And nope. It didn't work. cgroup-lite is in an
> endless loop right now :-)
>
> So I tried to put some echoes on the job. And the problem is.... (are
> actually) tail and awk
>
> awk is installed in /usr/bin/awk (which is a symlink to

Odd! I swear I had gone through the programs used in cgroups-mount
and found them all under /bin, but you're right.

> /etc/alternatives/akw, which is a symlink to /usr/bin/mawk). The same
> for tail.
>
> The for loop (for c in `tail -n +2 /proc/cgroups | awk....) at the end
> of /bin/cgroup-mount is never executed.
>
> I did a simple "cp /usr/bin/awk /bin/" and "cp /usr/bin/tail /bin/" and
> got cgroup-lite working.

I'll have to switch to using something other than awk.

> Now I'm wondering two things:
>
> 1) how did you get this to work with a separated /usr

Timing. /usr simply came up fast enough that awk was avaiable when I
needed it.

> 2) what others things are broken when /usr is on a separated partition.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cgroup-lite - 1.4

---------------
cgroup-lite (1.4) quantal; urgency=low

  * Don't use awk or tail, which are not in /bin. (LP: #989354)
 -- Serge Hallyn <email address hidden> Thu, 05 Jul 2012 11:21:39 -0500

[Sebastien Bacher (seb128)]

Could somebody with a setup to test the update check if it's indeed resolving the issue so it can be moved to -updates?

[Andreas Hasenack (ahasenack)]

I can't try the package from quantal, but I'm on precise and affected by this bug. I tried the 1.1.1 version from proposed and it fixed the problem for me, but I understand it could be just pure luck because awk and tail are outside of /usr.

[Serge Hallyn (serge-hallyn)]

Quoting Andreas Hasenack (<email address hidden>):
> I can't try the package from quantal, but I'm on precise and affected by
> this bug. I tried the 1.1.1 version from proposed and it fixed the
> problem for me, but I understand it could be just pure luck because awk
> and tail are outside of /usr.

SRU admins, please note that 1.1.2 is waiting with a full fix in Unapproved
(https://launchpad.net/ubuntu/precise/+queue?queue_state=1&queue_text=cgroup-lite)

[Andreas Hasenack (ahasenack)]

I'm sorry, but my previous statement wasn't completely accurate.

I had only checked that group was mounted, but when I tried to start a container it failed. It really only started working after I restarted cgroup-lite after a reboot.

For now I added this to /etc/rc.local before the exit 0 to workaround the issue, even with group-lite 1.1.1:

/etc/init.d/cgroup-lite restart

[Clint Byrum (clint-fewbar)]

Hello zoolook, or anyone else affected,

Accepted cgroup-lite into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cgroup-lite/1.1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please change the bug tag from verification-needed to verification-done. If it does not, change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Norberto Bensa (nbensa)]

Hello everyone!

works for me :-)

zoolook@lxc:~$ mount
/dev/mapper/vg_lxc-root on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
cgroup on /sys/fs/cgroup type tmpfs (rw,relatime,mode=755)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event)
/dev/mapper/vg_lxc-usr on /usr type ext4 (rw)
/dev/mapper/vg_lxc-var on /var type ext4 (rw)
/dev/mapper/vg_lxc-home on /home type ext4 (rw)
/dev/mapper/vg_lxc-boot on /boot type ext3 (rw)
zoolook@lxc:~$ apt-cache policy cgroup-lite
cgroup-lite:
  Instalados: 1.1.2
  Candidato: 1.1.2
  Tabla de versión:
 *** 1.1.2 0
        500 http://archive.ubuntu.com/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     1.1 0
        500 http://ar.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

[Andreas Hasenack (ahasenack)]

1.1.2 worked for me too. I was able to start a container right after a reboot.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cgroup-lite - 1.1.2

---------------
cgroup-lite (1.1.2) precise-proposed; urgency=low

  * Complete the fix for separate /usr (LP: #989354)
    - replace use of tail and awk with sed.
    - use echo instead of logger
 -- Serge Hallyn <email address hidden> Thu, 05 Jul 2012 11:46:30 -0500