cgmanager saturates one core

[Rolf Leggewie]

Over the past few days I've experienced the cgmanager binary saturating all CPU cycles on a single core on several occasions until I rebooted the machine. htop showed 100% of CPU cycles used.

Up-to-date trusty installation

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[ClashTheBunny (spam-mason)]

Here's my example:
root 287 27.5 0.4 35996 8932 ? Rs apr06 630:41 /sbin/cgmanager -m name=systemd
taken on April 08 around 5 am.

I'm on:
ii cgmanager 0.24-0ubuntu1 amd64 Central cgroup manager daemon
ii libcgmanager0:amd64 0.24-0ubuntu1 amd64 Central cgroup manager daemon (client library)
ii libcgmanager0:i386 0.24-0ubuntu1 i386 Central cgroup manager daemon (client library)

This process chews through one core on my machine.

I'm currently updating to -ubuntu3.

[Stéphane Graber (stgraber)]

Can you attach /var/log/upstart/cgmanager.log and if the daemon is still running out of control, trying to strace it with "strace -fF -p <pid>" would also be very useful to help figure this out.

Do you also have any idea of what you may have been doing on that machine when it started using all your CPU?

Oh and lastly, are you using lxc on that machine?

[Rolf Leggewie (r0lf)]

I cannot reliably reproduce this problem and do not really have any idea what might be triggering it. Maybe it's situations where resources (CPU, HD) are tight. While I have come across it a couple of times in the past I have yet to experience it again after reporting this ticket.

I have lxc installed but do not actively use it.

The content of the file you requested is not time-stamped, so I don't know if what's in there has anything to do with what I reported here. It might be from a completely different time. OK, here it is.

$ sudo cat /var/log/upstart/cgmanager.log
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:remove_main: Failed to remove /run/cgmanager/fs/none,name=systemd/user/1000.user/c2.session: Device or resource busy
cgmanager:remove_main: Failed to remove /run/cgmanager/fs/none,name=systemd/user/1000.user: Device or resource busy
cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///user (No such file or directory)
cgmanager:list_children_main: Could not determine the requested cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:remove_main: Failed to remove /run/cgmanager/fs/none,name=systemd/user/1000.user/c2.session: Device or resource busy
cgmanager:remove_main: Failed to remove /run/cgmanager/fs/none,name=systemd/user/1000.user: Device or resource busy
cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///user (No such file or directory)
cgmanager:list_children_main: Could not determine the requested cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager: Invalid path /run/cgmanager/fs/none,name=systemd///user (No such file or directory)
cgmanager:list_children_main: Could not determine the requested cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine the requestor cgroup
cgmanager:get_pid_cgroup_main: Could not determine...

[Stéphane Graber (stgraber)]

Yeah, that log isn't very helpful indeed... I was vaguely hoping for a massive logfile showing some kind of loop but yours appears to look mostly normal...

[Serge Hallyn (serge-hallyn)]

Are you still seeing this problem?

[Launchpad Janitor (janitor)]

[Expired for cgmanager (Ubuntu) because there has been no activity for 60 days.]

[Christian Ehrhardt  (paelzer)]

@Damiön - this bug was waiting to get better data to be able to tack down the issue.
Without that it can't be actions, one either needs working steps to reproduce or log-data that is enough to pinpoint the issue.
No one benefits form this bug being open but otherwise not going forward at all.

Therefore I'm setting it back to incomplete.
If you have more data that helps to debug or even better steps to reproduce then please share them here and then set it back to NEW for re-evaluation.

[Christian Ehrhardt  (paelzer)]

Found the extra data in bug 1759482
But nothing in there is enough for me (not a cgmanager expert) to find a root cause.
I'll mark the other bug a dup to this to be found from here.

@Serge - if you still look at this and see something in the trace please let us know.

[Launchpad Janitor (janitor)]

[Expired for cgmanager (Ubuntu) because there has been no activity for 60 days.]

[Rolf Leggewie (r0lf)]

I'm no longer on trusty, my machine is more beefy now and I've not run into it for a while. Damion was experiencing it fairly recently. If it is still a problem for you, then I suggest reopen bug 1759482 and reverse the dupe-relation.