cgconfig upstart job should start earlier and mount all available cgroup types by default

[urusha]

ubuntu 12.04 LTS, amd64
cgroup-bin 0.37.1-1ubuntu10

1. cgconfig upstart job should start earlier.
/etc/init/cgconfig.conf contains this:
  start on runlevel [2345]
but should contain this (like cgroup-lite.conf does)
  start on mounted MOUNTPOINT=/sys

Why it's needed? Imagine, I want cupsd to start with memory limit (cgroup called daemon-cups). To do this I edited configs:
/etc/cgconfig.conf
group daemon-cups {
    memory {
 memory.limit_in_bytes = 50M;
    }
}

 /etc/cgrules.conf
   root:/usr/sbin/cupsd memory daemon-cups

Now, after computer rebooted, I can see that daemon-cups cgroup is created but cupsd isn't in daemon-cups cgroup tasks. It appears there only after:
  service cups restart

That's because cups service started earlier than cgred service (and cgred service starts on started cgconfig).
Than imagine what if lxc will somehow start earlier than cgconfig? No, it won't start at all. So, I think this should be fixed.

2. cgconfig should mount all available cgroup types by default.
Default /etc/cgconfig.conf contains these:
mount {
 cpu = /sys/fs//cgroup/cpu;
 cpuacct = /sys/fs/cgroup/cpuacct;
 devices = /sys/fs/cgroup/devices;
 memory = /sys/fs/cgroup/memory;
 freezer = /sys/fs/cgroup/freezer;
}

What if I want to use LXC on the same machine and one container's config contains something like this:
  lxc.cgroup.cpuset.cpus = 1
Container will not start, because this cgroup type is not mounted by default with cgroup-bin package.
I think, the best way to fix this is to mount available cgroup fsystems via upstart job (the way cgroup-lite does it). Another way is to change default /etc/cgconfig.conf (add mountpoints). But I would prefer the first solution (it's kernel independent).

3. After all these I'm asking you what is the reason of separation cgroup-bin and cgroup-lite into different packages. Why not to just disable cgrulesengd, cgconfigparser by default (using /etc/default somehow)? For example, if I'm using LXC, I will likely be using cgget/cgset or something. And even if won't be using it what's wrong with that these binaries are in my /usr/bin (100Kb)?

P.S. cgconfig.conf : cpu = /sys/fs//cgroup/cpu; I believe double slashes isn't needed here.

Thanks, hope it'll be fixed in LTS.

[Serge Hallyn (serge-hallyn)]

Thanks for submitting this bug. I agree with your analysis.

The reason for the separate cgroup-lite package is that simply disabling the broken libcgroup tools by default, but still installing them, would (a) be harder to get into main (and not justifiable in my opinion) and (b) still look like an endorsement of the tools, when we know they are broken (for instance when used with libvirt). Plenty of new users would re-enable them, break libvirt, and file bugs.

If you are using lxc and wish to manipulate cgroups with a tool, the preferred way would be to use the lxc-cgroup command. I do believe (having just checked) there is a bug in lxc-cgroup in precise, but that should be fixed and should be a simple fix.

[Serge Hallyn (serge-hallyn)]

My agreement was with point 1. I don't agree with point 2, as that is a diversion from the libcgroup upstream and debian package. They do not simply want all cgroups mounted separately by default. Rather, they want it to be very configurable. So I wouldn't object strongly to adding the extra cgroups to the configuration file (though I think that change should simply be done through debian), but I think it is wrong to change how the cgroups are mounted.

The libcgroup package was never simply a package to mount all cgroups and provide set/get cgroup settings. It has a specific purpose (which it cannot quite meet in a race-free way) of flexible, configurable cgroup management by users, groups, and programs.

Perhaps what you'd actually like is to add a simple cgroup set/get program to cgroup-lite.

I am going to mark this bug incomplete while awaiting your feedback on the following proposal: I'd like to change the title of this bug to simply be 'cgconfig upstart job should start earlier', and make that change and p and q. (Then merge debian's new version into q). Point 2 I'd like to be handled through debian.

[urusha (urusha)]

I see you. The critical issue is point 1, while point 2 is more a configuration issue. Now after your explanation I agree with your opinion about point 2. A better way is to add mount points (cpuset,blkio) to the default configuration file. What should I do for it then?

Well, I'm not against changing the title if it's needed.

The main purpose for me is to run daemons with limits. Upstart doesn't handle cgroups and cgroup limits at the moment, so the final optimal way I found is:
a) use cgconfig.conf to set limits
b) use service.override and cgexec to run daemon in the special cgroup. cups example:
  exec cgexec -g cpuset:daemon-cups /usr/sbin/cupsd -F

cgred could also be used, but if you use lxc on the same machine it affects all lxc container's daemons (cups) too and I don't need such limits. Anyway point 1 of this bug report is needed for this method to work too (cgconfig should start before cups).

Could you give me a link about libcgroup+libvirt problems?

Thanks.

[Serge Hallyn (serge-hallyn)]

Sorry for the slow response, I didn't see your response in my email somehow.

See https://bugs.launchpad.net/ubuntu/+source/libcgroup, and in particular https://bugs.launchpad.net/ubuntu/+source/libcgroup/+bug/827419 for libcgroup+libvirt problems.

[Serge Hallyn (serge-hallyn)]

actually as discussed on irc, start on mounted MOUNTPOINT=/sys won't work because it doesn't guarantee that the filesystem is sufficiently set up for the libcgroup binaries to be found or run.

we could do start on filesystem, but many other daemons already start then so that would not suffice for this bug.

[urusha (urusha)]

Hi. Are you talking about binaries in /usr/*bin directories? If so, then you mean that system can be booted without /usr(bin,sbin) somehow? If so, I'm very surprised about that ability (I see it can be used somehow for ro-mounted /usr and, maybe, with embedded devices or small-size storages), but anyway:
a) Is this technique used such widely, then we can't change defaults? (it's users can make cgconfig.override too...)
b) why then not to move binaries into /bin, /sbin? Despite the fact this package is buggy, it's very system-internal, isn't it?

If I see you wrong, please explain.
Thanks

[Serge Hallyn (serge-hallyn)]

> Hi. Are you talking about binaries in /usr/*bin directories? If so, then you mean that system can be booted without /usr(bin,sbin) somehow?

No. I mean that the event 'mounted MOUNTPOINT=/sys' can happen before /usr is mounted.

[urusha (urusha)]

Well, I mean exactly the same, my english isn't well:). And then I asked - are there many people using /usr on another partition than /? If no - why not to change defaults? If yes, is it a problem to just move binaries to /bin, /sbin?

[urusha (urusha)]

And I've just thought: is it possible to implement something like "mounted MOUNTPOINT=/sys and mounted MOUNTPOINT=/usr"? Does upstart support nonexistent mount points (not fails if there is no /usr mount point)?