console-kit-daemon using a lot of cpu

Bug #284229 reported by Mirar
98
This bug affects 7 people
Affects Status Importance Assigned to Milestone
ConsoleKit
Fix Released
High
consolekit (Ubuntu)
Fix Released
Medium
Unassigned
Hardy
Invalid
Undecided
Unassigned
Intrepid
Fix Released
Undecided
Unassigned

Bug Description

The use of pam-ck-connector means that consolekit registers a session for every cron or ssh session
started. Registering a session involves creating a dbus proxy. Internally to glib these are stored in a list,
and only removed when unref'd, but the code fails to do that. If many sessions are opened
(high frequency cron for instance) then the list grows in length, and operations that traverse the list take
increasing amounts of time. This can lead to consolekit taking up large amounts of CPU time.

The fix is simply to unref the dbus proxy when we are removing the session. It is then removed from the
internal list and causes no problems. This fix was deployed in jaunty on 2009/01/27.

The patch for Intrepid is available in a later comment.

TEST CASE:
  1. With an ssh server installed on the machine and keys set up so you can log in without password run
         do while `true`; do ssh localhost echo '$$'; done
      and leave it running overnight. It should lead to console-kit-daemon using a high percentage of the CPU.

The regression potential should be low as unrefing something that is no longer used when the session
is being removed should not be a problem.

=== Original Report ===

Binary package hint: consolekit

Since the last reboot, a strange and to me unknown process, console-kit-dae, has been using up more and more cpu.

Right now it uses on average 10.25% cpu over 5 minutes, steadily and linearly (!) growing in cpu usage and have been using up 7h 36 minutes CPU over 7 days.

This doesn't seem to be quite normal.

intrepid, some week old since last update.

Revision history for this message
Mirar (launchpad-sort) wrote :

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
25676 root 20 0 178m 17m 1560 R 71 0.9 457:02.40 console-kit-dae

Revision history for this message
Mirar (launchpad-sort) wrote :

Seems repeatable, at least:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28634 root 20 0 179m 17m 1380 S 55 0.9 453:26.99 console-kit-dae

Since noone seems interested in this, I will turn off running of console-kit-daemon for now.

Revision history for this message
James Westby (james-w) wrote :

Hi,

Could you please grab an strace of the process?

  sudo strace -f -o /tmp/consolekit-strace.txt -p <PID>

where PID is taken from top or similar, 28634 in your
last comment.

Once you have captured a couple of minutes worth ctrl-c the strace
and attach /tmp/consolekit-strace.txt to the bug.

Thanks,

James

Revision history for this message
Mirar (launchpad-sort) wrote :

I restarted the process now, so probably everything's normal...

17226 root 20 0 168m 3376 1632 S 0 0.2 0:15.08 console-kit-dae

It usually takes at least a few days before it uses a noticable amount of cpu. It uses 0.07% cpu on average right now.

Revision history for this message
James Westby (james-w) wrote : Re: [Bug 284229] Re: console-kit-daemon using a lot of cpu

On Sat, 2008-10-25 at 17:23 +0000, Mirar wrote:
> I restarted the process now, so probably everything's normal...
>
> 17226 root 20 0 168m 3376 1632 S 0 0.2 0:15.08 console-
> kit-dae
>
> It usually takes at least a few days before it uses a noticable amount
> of cpu. It uses 0.07% cpu on average right now.

Thanks,

Next time you see it use a lot of CPU could you please grab another
strace for comparison?

Thanks,

James

Revision history for this message
Mirar (launchpad-sort) wrote :
Download full text (3.9 KiB)

It's not using a lot of cpu yet, but it's up to an average of 3% now, 1.8 cpu-seconds per minute on average.

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17226 root 20 0 175m 9684 1748 R 16 0.5 53:39.34 console-kit-dae

From the strace and the way top keeps showing 0% cpu I would say it uses all those cpu-seconds are used in the beginning of the minute... yep.

# top -b | grep console
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:40.10 console-kit-dae
17226 root 20 0 175m 9684 1748 R 25 0.5 53:40.85 console-kit-dae
17226 root 20 0 175m 9684 1748 S 18 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 175m 9684 1748 S 0 0.5 53:41.39 console-kit-dae
17226 root 20 0 ...

Read more...

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

I observe the same thing in 8.10 release. console-kit-daemon gets a steady 4% of CPU on a 3.0GHz Core 2 Duo. Minute-long strace attached.

Revision history for this message
Mirar (launchpad-sort) wrote :

Which reminds me, mine is now up to 6.5% cpu (almost 4 cpu-seconds a minute). Attaching new strace, run at this moment:

17226 root 20 0 177m 13m 1648 S 0 0.7 176:55.62 console-kit-dae
17226 root 20 0 177m 13m 1648 S 62 0.7 176:57.50 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 176:57.50 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 176:57.50 console-kit-dae
...
17226 root 20 0 177m 13m 1648 S 0 0.7 176:57.50 console-kit-dae
17226 root 20 0 177m 13m 1648 R 93 0.7 177:00.34 console-kit-dae
17226 root 20 0 177m 13m 1648 R 96 0.7 177:03.22 console-kit-dae
17226 root 20 0 177m 13m 1648 S 98 0.7 177:06.16 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 177:06.16 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 177:06.16 console-kit-dae
...
17226 root 20 0 177m 13m 1648 S 0 0.7 177:06.16 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 177:06.16 console-kit-dae
17226 root 20 0 177m 13m 1648 R 3 0.7 177:06.24 console-kit-dae
17226 root 20 0 177m 13m 1648 S 29 0.7 177:07.12 console-kit-dae
17226 root 20 0 177m 13m 1648 R 1 0.7 177:07.14 console-kit-dae
17226 root 20 0 177m 13m 1648 S 62 0.7 177:09.00 console-kit-dae
17226 root 20 0 177m 13m 1648 S 0 0.7 177:09.00 console-kit-dae
...

Revision history for this message
James Westby (james-w) wrote :

Hi,

Thanks, you have some dbus activity, and it appears to be opening
and closing sessions.

Could you also grab some output from "dbus-monitor --system" so that
we can watch the traffic (you might want to screen the output to check
for private information)

Also your /var/log/ConsoleKit/history file would be useful.

Thanks,

James

Revision history for this message
In , Jeffrey Baker (jwbaker) wrote :

console-kit-daemon eventually uses 100% on any system that is up for a sufficiently long time, which can be as short as days. The problem seems to be that console-kit-daemon is leaking sessions, and has some kind of linear or superlinear algorithm for adding or removing a session. This is accompanied by a steady increase in the process size.

I'm using consolekit 0.2.10 from Ubuntu (0.2.10-1ubuntu9, Ubuntu 8.10, x86).

You can easily reproduce this for yourself, as follows:

while `true`; do ssh localhost 'echo $$ && exit'; done;

Assuming you have SSH keys which allow you to login non-interactively. On my system, which as a 3.0GHz dual-core processor, console-kit-daemon started out using less than 1% CPU and less than 2MB of memory, but after ten minutes of this test it's using 19% CPU and has 3.5MB of memory. When I killed it earlier it had used 800 CPU minutes and had 15MB of memory.

Anybody who has a lot of users or cron jobs that fire every minute should be able to see this bug in practice.

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

The way I read it, sessions are being leaked into console-kit-daemon, and every time a session is added or removed console-kit-daemon does something that's either O(N) or O(N^2) or worse. I'm up to Session #31399 and every time a session is added or removed, c-k-d eats another 4 or 5 CPU seconds (100% utilization for several seconds). This would also account for c-k-d's ever-increasing memory size.

From what I can tell, the extra sessions are due to cron. I have some cron jobs that fire every minute, so they are creating and destroying sessions pretty frequently.

Which means that I'm leaking a session every minute. The superlinear big-O explains why c-k-d is taking an increasingly large fraction of CPU time as the system stays up longer. It was 4.1% yesterday and it's up to 4.8% now. It takes me 2-3 seconds to sudo because c-k-d is spending so long with its head up its ass.

I suspect that this would hit anyone with minutely cron jobs.

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

Here's a really easy way to reproduce this problem. Assuming you have an ssh keypair which allows you to login on your own machine, do this:

while `true`; do ssh localhost exit; done;

console-kit-daemon will runaway with the CPU eventually.

In my opinion this is a very serious problem for anybody running Ubuntu who has a lot of users or who has cron jobs.

Changed in consolekit:
status: Unknown → Confirmed
Jeffrey Baker (jwbaker)
Changed in consolekit:
status: New → Confirmed
Revision history for this message
Mirar (launchpad-sort) wrote :

This seems plausible. I have a 5 minute interval cron job (munin). Still need my outputs?

Revision history for this message
James Westby (james-w) wrote :

Hi,

I have tried the ssh loop to try and trigger this, and console-kit-daemon uses a few percent
CPU while it is running, but then immediately falls back to zero when it is killed. Do I need
to wait longer for this to become evident? I am up to session 1096.

Thanks,

James

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

Ditto for me if I leave the PC/Lap Top on for some time when I return to it the fan is going full tilt and it's all due to the 'console-kit-dae' process making my CPU work alot. What is its purpose because if I kill the process there are seemingly no ill effects, can I just bin it and suffer no ill effects?

Revision history for this message
Ryan Davies (iownsu) wrote :

I get the same, except in the server flavour of Ubuntu.

I'll be sitting here, and every 10-15 minutes, the CPU jumps to 100%, the fan goes full tilt. I thought it might have been alot of dust, but after having htop running for a bit, i notice console-kit-daemon is the culprit.

Revision history for this message
Jonathan Hudson (jh+lpd) wrote :

Similarly, I have a headless server that does all its work from cron. After a while (maybe a week), it's unusable because console-kit-daemon is taking 99% of the CPU.

Removal seems unwise, as it offers to remove lots of useful packages, so I suppose the answer is another cron job that kills console-kit-daemon with extreme prejudice.

Somewhat surprised that a month old server crippling bug is still of 'unknown' importance; in my case it's severe.

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

I haven't seen this in a wee while, maybe it's because I have started to shutdown instead of serializing to RAM but I'm not too sure.

"Somewhat surprised that a month old server crippling bug is still of 'unknown' importance; in my case it's severe"

This is why Debian is still such a good choice for the server if you don't need up-to-date packages.

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

I profiled this piece of junk. Console kit daemon spends more than 94% of its time in g_slist_find() at the top of this stack:

#0 0xb7f17db9 in IA__g_slist_find (list=0xa164ae0, data=0xa033210)
    at /build/buildd/glib2.0-2.18.2/glib/gslist.c:574
#1 0xb800c5a9 in dbus_g_proxy_manager_filter (connection=0x9ed2f00, message=0xa1ee990, user_data=0x9ed5818)
    at dbus-gproxy.c:1259
#2 0xb7fd20d5 in dbus_connection_dispatch () from /lib/libdbus-1.so.3
#3 0xb80035fd in message_queue_dispatch (source=0x9ed33d8, callback=0, user_data=0x0) at dbus-gmain.c:101
#4 0xb7ef86f8 in IA__g_main_context_dispatch (context=0x9ed2070)
    at /build/buildd/glib2.0-2.18.2/glib/gmain.c:2144
#5 0xb7efbda3 in g_main_context_iterate (context=0x9ed2070, block=1, dispatch=1, self=0x9ecbc68)
    at /build/buildd/glib2.0-2.18.2/glib/gmain.c:2778
#6 0xb7efc2c2 in IA__g_main_loop_run (loop=0x9edf1e0) at /build/buildd/glib2.0-2.18.2/glib/gmain.c:2986
#7 0x0804e586 in main (argc=1, argv=0xbff49834) at main.c:373

My suspicion is that c-k-d is adding dbus handlers and not removing them, so the list of proxies grows without bounds, and it must be walked every time a dbus message arrives. That's just a hunch though, and I haven't found the exact line of code where the root of the problem lies.

Revision history for this message
In , Jeffrey Baker (jwbaker) wrote :

The downstream has developed a fix for this issue at http://launchpadlibrarian.net/19884292/11-unref-dbus-proxy

Revision history for this message
Marcin Woloszyn (dervsh) wrote :

As for me it only concerns Ubuntu 8.10 Server, i386, on quad core Xeon,

right now, it results:
  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
30189 root 39 19 38672 22m 1624 R 100 2.3 9358:44 console-kit-dae
    7 root 15 -5 0 0 0 S 2 0.0 389:38.13 ksoftirqd/1
   13 root 15 -5 0 0 0 S 0 0.0 309:39.80 ksoftirqd/3
30144 root 30 10 13264 6748 2248 S 0 0.7 189:51.44 traf.py
    4 root 15 -5 0 0 0 S 0 0.0 30:17.34 ksoftirqd/0
   10 root 15 -5 0 0 0 S 0 0.0 30:17.23 ksoftirqd/2
 4369 postgres 20 0 42084 26m 25m S 0 2.6 15:30.96 postgres
 2351 root 15 -5 0 0 0 S 0 0.0 12:16.94 kjournald
[...]

sorted by cpu time, I used renice 19 on console-kit-daemon.
at pstree console-kit-daemon results:

     ├─console-kit-dae───63*[{console-kit-dae}]

machine runs middle sized postgres db and shapes traffic for 100Mbit/s ethernet.
Console-kit altrough dominates at cpu load. Looking forward any, even temporary solution.
Is it safe to kill console-kit process? or at least - send HUP to process pid?

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

"Is it safe to kill console-kit process?"

There are no side effects for me.

Revision history for this message
James Westby (james-w) wrote :

Jeffrey, thanks for the investigation. I agree with your assesment.

It seems like the attached patch may be enough to solve this. It is completely
untested though. Would someone who sees this issue be able to
test it?

Thanks,

James

Changed in consolekit:
importance: Undecided → Medium
Revision history for this message
Jeffrey Baker (jwbaker) wrote :

I think you nailed it, James. With your patch, c-k-d uses a steady amount of CPU over time (about 2% with a high login load). It's still leaking memory slowly, but I now assume that's a separate bug.

+1 from me on your patch. Note that it applied with an offset to the Intrepid version of c-k-d. I didn't check against the SVN sources.

Thanks a lot,
Jeff

Revision history for this message
James Westby (james-w) wrote : Re: [Bug 284229] Re: console-kit-daemon using a lot of cpu

On Sun, 2008-11-23 at 19:37 +0000, Jeffrey Baker wrote:
> I think you nailed it, James. With your patch, c-k-d uses a steady
> amount of CPU over time (about 2% with a high login load). It's still
> leaking memory slowly, but I now assume that's a separate bug.

Thanks for testing, glad to know that it seems to help.

I would appreciate testing by someone else, as I was unable to reproduce
the problem in the first place, and I'm not totally sure that the patch
is correct. Would someone else be able to test the package?

Thanks,

James

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

I think you would be able to reproduce it with just slightly more testing time. 1000 logins probably isn't enough to reveal the problem. I just do while `true`; do ssh localhost echo '$$'; done and let it run for an hour, or overnight, and then I believe you'll see it. There's no special reason why this would affect a subset of systems. I see it on bone stock Ubuntu installations on perfectly ordinary computers. The only difference is that some operators exercise the login subsystem a lot more than others.

Revision history for this message
HighInBC (highinbc) wrote :

A graph of how this problem effects my CPU over the course of 11 days uptime.

console-kit-daemon was using 100% of my CPU most of the time when I finally rebooted.

I have a cronjob that runs every minute so I am spawning 1440 logins a day.

Revision history for this message
Benoît Rouits (brouits) wrote :

i had the same problem, having console-kit-daemon usgin about 10% of CPU, and growing.
I saw (thx to "lsof -c console-kit") that console-kit-daemon wrote very often (about every 0.2 sec!) to his logfile (/var/log/ConsoleKit/history) an apparent login/logout from user 1000 (which is myself!) while i did not logon/off:

1227922049.237 type=SEAT_SESSION_ADDED : seat-id='Seat1' session-id='Session3358' session-type='' session-x11-display='' session-x11-display-device='' session-display-device='/dev/???' session-remote-host-name='' session-is-local=TRUE session-unix-user=1000 session-creation-time='2008-11-29T01:27:29.171247Z'
1227922049.449 type=SEAT_SESSION_REMOVED : seat-id='Seat1' session-id='Session3358' session-type='' session-x11-display='' session-x11-display-device='' session-display-device='/dev/???' session-remote-host-name='' session-is-local=TRUE session-unix-user=1000 session-creation-time='2008-11-29T01:27:29.171247Z'

[and so on...]

i do not have any crontab for uid 1000. So i don't know what made such a stress on console-kit-daemon.

Revision history for this message
James Westby (james-w) wrote :

On Sat, 2008-11-29 at 01:46 +0000, Ben wrote:
> i had the same problem, having console-kit-daemon usgin about 10% of CPU, and growing.
> I saw (thx to "lsof -c console-kit") that console-kit-daemon wrote very often (about every 0.2 sec!) to his logfile (/var/log/ConsoleKit/history) an apparent login/logout from user 1000 (which is myself!) while i did not logon/off:
>
> 1227922049.237 type=SEAT_SESSION_ADDED : seat-id='Seat1' session-id='Session3358' session-type='' session-x11-display='' session-x11-display-device='' session-display-device='/dev/???' session-remote-host-name='' session-is-local=TRUE session-unix-user=1000 session-creation-time='2008-11-29T01:27:29.171247Z'
> 1227922049.449 type=SEAT_SESSION_REMOVED : seat-id='Seat1' session-id='Session3358' session-type='' session-x11-display='' session-x11-display-device='' session-display-device='/dev/???' session-remote-host-name='' session-is-local=TRUE session-unix-user=1000 session-creation-time='2008-11-29T01:27:29.171247Z'
>
> [and so on...]
>
> i do not have any crontab for uid 1000. So i don't know what made such a
> stress on console-kit-daemon.
>

Hi,

Could you please capture some output of "dbus-monitor --system" while
this is happening please.

The session-display-device='/dev/???' is odd.

Thanks,

James

Revision history for this message
seen (seven-idapted) wrote :

root# top -b |grep dae
 3575 root 20 0 7904 1972 1432 R 68.2 0.2 562:28.32 console-kit-dae
 3575 root 20 0 7904 1972 1432 R 66.2 0.2 562:30.31 console-kit-dae
 3575 root 20 0 7904 1972 1432 R 66.6 0.2 562:32.31 console-kit-dae
 3575 root 20 0 7904 1972 1432 R 66.1 0.2 562:34.30 console-kit-dae
 3575 root 20 0 7904 1972 1432 S 66.7 0.2 562:36.30 console-kit-dae
 3575 root 20 0 7904 1972 1432 R 66.6 0.2 562:38.30 console-kit-dae

Revision history for this message
seen (seven-idapted) wrote :

top - 10:27:55 up 13 days, 22:02, 2 users, load average: 11.18, 11.90, 12.33
Tasks: 76 total, 5 running, 71 sleeping, 0 stopped, 0 zombie
Cpu(s): 77.2%us, 22.8%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1229012k total, 1197872k used, 31140k free, 141624k buffers
Swap: 1048568k total, 351856k used, 696712k free, 252108k cached

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

This is still occurring for me, there appears to be a patch in the chatter but no updated packages. Is their some way I can assist in getting this finally corrected?

Revision history for this message
André Cruz (andrefcruz) wrote :

This is also still occurring to me.

Revision history for this message
vak (khamenya) wrote :

similar happens to me.

% sudo strace -f -o /tmp/consolekit-strace.txt -p 5837
Process 5898 attached with 62 threads - interrupt to quit
Ctrl-C
Process 5837 detached

and here goes output after 1 minute of stracing:

5854 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5855 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5856 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5857 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5858 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5859 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5860 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5861 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5862 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5863 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
5864 ioctl(11, VIDIOC_S_COMP or VT_WAITACTIVE <unfinished ...>
[...and it goes so further...]

Revision history for this message
Ulrich Schreiner (ulrich-schreiner) wrote :

i had the same problem with consolekit. after a few days consolekit-daemon had about >90% of cpu.

then i realized that in my "auth.log" that every 5 seconds i get this entries:

Jan 4 06:24:36 usc-desktop sudo: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/x2gopgwrapper listsessionsroot usc-desktop
Jan 4 06:24:36 usc-desktop su[22542]: Successful su for postgres by root
Jan 4 06:24:36 usc-desktop su[22542]: + ??? root:postgres
Jan 4 06:24:36 usc-desktop su[22542]: pam_unix(su:session): session opened for user postgres by (uid=0)
Jan 4 06:24:36 usc-desktop su[22542]: pam_unix(su:session): session closed for user postgres

so i disabled the "x2goserver" and voila ... consolekit-daemon is really handy now :-)

i don't know where the problem exactly is; i only realized that x2goserver drives consolekit-daemon crazy. is it a bug or misconfiguration in x2go or in consolekit? i really don't know ...

Revision history for this message
vak (khamenya) wrote :

I agree with user "jh". For me the urgency is severe or high. I could write a script to kill these processes, but this may kill healthy activity too ... So, until a workaround is not clear for me it is severe too. Also, I am quite worrying about the age of this bug in this unsolved state. This looks for me like with current urgency we'll hardly get it solved in somewhat reasonable time.

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

Hi Ulrich, I have had a look at my auth file and don't see any mention of x2goserver yet still see the consolekit problem however with less frequency than I used to. I think you might have a seperate issue that inflames the consolekit bug.

Sorry, about to go off topic but a little interesting all the same:

I would have thought that all you guys running a server would be using the 8.04 LTS edition which doesn't have this problem and is more intended for long term deployment (On the desktop this issue is a mere hindrance). That said I would have though the Ubuntu web servers would have been on 8.04 and not Dapper 6.04 as looking at Netcraft:

http://toolbar.netcraft.com/site_report?url=http://www.ubuntu.com

it reports the version in use is Apache/2.0.55 which is the version used in Dapper 6.04:

http://packages.ubuntu.com/dapper/web/apache2

Like I said off topic but interesting none the less.

Revision history for this message
glenstewart (glen-stewart) wrote :

I wrote this Monit entry to try to bring console-kit-daemon under control, but it hasn't seemed to be effective, as the console-kit-daemon CPU use seems to fluctuate as it ramps up to seize all CPU. As this happens, I lose interactive control of my X session, even though running on a dual-CPU system.

check process console-kit-daemon with pidfile /var/run/console-kit-daemon.pid
   start program = "/usr/bin/ck-launch-session"
   stop program = "killall -9 console-kit-daemon"
   if 5 restarts within 5 cycles then timeout
   if cpu usage > 3% for 5 cycles then alert

I agree - the urgency of this is no less than HIGH, and needs quick attention.

Revision history for this message
Ulrich Schreiner (ulrich-schreiner) wrote :

hi,

on my system (8.10 desktop) i installed a x2go and this server opens every 5 secons a new session (as my posting above shows). and as the pam_ck_connector is installed to (/etc/pam.d/common-session), this session is registered with the consolekit-daemon --> every 5 seconds a new session is registered with consolekit-daemon. i don't know if this session is deregistered. i only see, that if i disable this "session inflation", consolekit-daemon also is ok.

perhaps some of you have another daemon running which also opens a lot of sessions in the background. so don't look if you have a x2go, look for something which creates very much sessions. this was my "main problem".

perhaps it helps to disable pam_ck_connector? i don't know if this is needed ...

only my 2 cents ...

Revision history for this message
In , James Westby (james-w) wrote :

Hi,

There have now been two separate confirmation that my patch
improves the situation.

Please consider applying it, or let me know what the preferred
fix would be if not.

Thanks,

James

Revision history for this message
In , James Westby (james-w) wrote :

Created an attachment (id=21900)
Unref the dbus proxy when finalizing the session

Revision history for this message
ichudov (igor-chudov) wrote :

I have written a shell function that downloads source of consolekit, some related libraries (I am not sure if you might need some more than I download), makes a patch, compiles and installs consolekit and restarts it. After this, I can ssh to this box in a loop forever and the CPU use does not grow. See attachment.

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

I was taking a look at the script and all looked well at a glance (usual checks for 'rm /*' jazz)

and when I run it I got './consolekit-fn.sh: 45: Syntax error: end of file unexpected (expecting ")")'

but there isn't a missing parenthesis, at least not picked up by the highlighting in kate! Any ideas 'ichudov'

Thanks & Regards

Revision history for this message
ichudov (igor-chudov) wrote :

This is a bash function to be run by bash.

Save it to a file, say func.sh. Then in your _bash_ session, say

source func.sh

then run command

ROOT_FixConsoleKit

Revision history for this message
James Westby (james-w) wrote :

Hi ichudov,

Thanks for testing the patch, with two confirmations of it working I'm
comfortable requesting that it be uploaded.

Thanks,

James

Revision history for this message
James Westby (james-w) wrote :

Sponsors, please review for inclusion in to Jaunty.

Thanks,

James

Revision history for this message
ichudov (igor-chudov) wrote :

James, thanks a lot, this was the most obnoxious Intrepid issue for me, I am glad that we are moving with it.

Revision history for this message
David Cournapeau (david-ar) wrote :

Is there a reason this is not backported to Intrepid ? I don't have admin privileges on my box, and this makes Ubuntu unbearable. After a few hours, I have to restart X, or worse rebooting my machine.

Revision history for this message
Benjamin Lauret (benlauret2000) wrote :

For me the patch solve the issue. I need 3 days to have it taking 100% of the CPU. It's been running smoothly over the the last 4 days.

Revision history for this message
jasherai (phatforge) wrote :

+1 on the backport. It would be really useful for some of the servers I'm running.

Revision history for this message
Mirar (launchpad-sort) wrote :

Is certainly should be backported. I've solved it by killing it regularly though:

# crontab -l -u root
# m h dom mon dow command
0 10,20 * * * killall console-kit-daemon
...

Revision history for this message
Martin Pitt (pitti) wrote :

Did people experience this on Hardy (8.04) as well, or just in Intrepid (8.10)?

Changed in consolekit:
status: New → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package consolekit - 0.3-0ubuntu3

---------------
consolekit (0.3-0ubuntu3) jaunty; urgency=low

  * Unref the dbus proxy when finalizing the session object, so that
    they are not leaked. (LP: #284229)
    - When leaked they would stay in the list of active proxies, which
      is iterated over during some proxy operations. This would lead to
      a dramatic slowdown and CPU usage spike if a lot of sessions were
      created and destroyed.

 -- James Westby <email address hidden> Mon, 12 Jan 2009 16:09:17 +0000

Changed in consolekit:
status: Confirmed → Fix Released
Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

"Did people experience this on Hardy (8.04) as well, or just in Intrepid (8.10)?"

Just intrepid for me and looking at the report it was logged around the release of intrepid.

Revision history for this message
glenstewart (glen-stewart) wrote :

This only occurred with Intrepid.

As the bug can render a server unusable due to CPU load, the fixed package mentioned for Jaunty should be backported to Intrepid. I consider it URGENT to address severe performance problems.

Revision history for this message
Martin Pitt (pitti) wrote :

Approved for SRU, does someone want to drive this?

Changed in consolekit:
status: Incomplete → Invalid
status: New → Confirmed
Revision history for this message
James Westby (james-w) wrote :

I can do the legwork. I'll probably again need some assistance verifying
the fix though, as I have yet to reproduce it.

Thanks,

James

Revision history for this message
Travis Tabbal (travis-tabbal) wrote :

I'm willing to test the fixed package. I can make it happen reliably on my systems.

Revision history for this message
Sean Covel (sean-covel) wrote :

Ubuntu Intrepid Ibex. Fresh install. No problems for a few months. Added Munin and Monit. Now console-kit-daemon takes over in a couple weeks of uptime. Must be related to the login/logout every 5 min. that Munin does. Makes it happen much faster.

Linux gordon.home.com 2.6.27-9-server #1 SMP Thu Nov 20 22:53:41 UTC 2008 i686 GNU/Linux

Revision history for this message
goto (gotolaunchpad) wrote :

Running `pam-auth-update` allows you to remove using ConsoleKit as a session manager. I then killed off console-kit-daemon and it has not restarted since (lots of remote cron tasks).

Use of ConsoleKit on a box that gets connected to at a minimum of once a minute over a 2 month period has lead to a process that slows down ssh login to take over 30 seconds, which in turn creates problems.

This was on an EC2 ubuntu image with a minimal install.

Revision history for this message
vbonline (volker-buescher) wrote :

Using ubuntu 8.10-i386 server with Cacti / Munin / Horde each of them logging in at least once every 5 minutes, so I'm really bitten by this bug. I have to restart console-kit-daemon once a week to get my system responsible again.

To make it clear: This is a MAJOR bug and it is absolutly UNACCEPTABLE to have a server system rendered unresponsive ONCE a WEEK. If I want to restart services on a weekly basis I can as well use software from redmond...

Now: When is a patch to be expected for intrepid?

Alternative: Give a COMPLETE doc on how to apply the patch. The patched console-kid-daemon with the patch out of this thread won't compile b/c there is no X11 libs available. Guess what, this is a server without x11. What to do?

Fighting with this bugs a few weeks now and my patience is running out....

Revision history for this message
Jonathan Hudson (jh+lpd) wrote :

I found that

# chmod -x /usr/sbin/console-kit-daemon

provided a good enough fix for my headless, X-less box that was otherwise afflicted by this bug.

-jh

Revision history for this message
vbonline (volker-buescher) wrote :

To be honest: Rendering c-k-d unexecutable or uninstalling c-k-d is the wrong way to remedy this situation. Having c-k-d around is a Good Thing (tm) by definition...

But having a patch since 11/28/08 (77 days) and NOT being able to "apt-get upgrade" this bug into oblivion is the problem here...

Revision history for this message
Mirar (launchpad-sort) wrote :

Frankly, I can't really find out which problem it's supposed to solve, maybe you can elaborate? Google searches only turn up problems, and

# man console-kit-daemon
No manual entry for console-kit-daemon

doesn't help either. (But there doesn't seem to be many manpages around these days.)

I'm sticking to my cron-killall-solution for now. Since it's a problem with data structures leaking it seems to work fine.

Revision history for this message
ichudov (igor-chudov) wrote :

I also cannot believe that this bug has not even been patched in the repos. Patch has been posted and verified weeks ago. What is going on?

Revision history for this message
David Taylor (me-davidandrewtaylor) wrote :

I thought I may be able to just use the Debian version of this package instead as this just seems to be an Ubuntu thing but I can't seem to find a Debian alternative. Maybe it's only Ubuntu who use it.

Still I'll give Suse a go as there are a number of bugs I have that aren't being fixed but I remember the last time I used a RPM distribution and it was hopeless but then again maybe they have caught up.

Revision history for this message
Jeffrey Baker (jwbaker) wrote :

I think the main problem is that the upstream developer seems to have abandoned the project.

Revision history for this message
In , James Westby (james-w) wrote :

Hi,

This patch is now being shipped in the development release of Ubuntu,
with no reported problems. I am currently preparing an update for Ubuntu
8.10 as there are plenty of users reporting the problem there.

Thanks,

James

James Westby (james-w)
description: updated
Revision history for this message
James Westby (james-w) wrote :

Here is the debdiff for Intrepid.

Thanks,

James

Revision history for this message
Martin Pitt (pitti) wrote :

Accepted into intrepid-proposed, please test and give feedback here. Thank you!

Changed in consolekit:
status: Confirmed → Fix Committed
Revision history for this message
In , William Jon McCann (william-jon-mccann) wrote :

Sorry for the long delay. Thanks for the patch! I've applied this to master now.

Revision history for this message
Wolfgang Tremmel (launchpad-garf) wrote :

Installed it and it solved the problem for me.

Changed in consolekit:
status: Confirmed → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package consolekit - 0.2.10-1ubuntu10

---------------
consolekit (0.2.10-1ubuntu10) intrepid-proposed; urgency=low

  * Unref the dbus proxy when finalizing the session object, so that
    they are not leaked. (LP: #284229)
    - When leaked they would stay in the list of active proxies, which
      is iterated over during some proxy operations. This would lead to
      a dramatic slowdown and CPU usage spike if a lot of sessions were
      created and destroyed.

 -- James Westby <email address hidden> Tue, 10 Feb 2009 14:53:11 +0000

Changed in consolekit:
status: Fix Committed → Fix Released
Revision history for this message
vak (khamenya) wrote :

Martin Pitt wrote on 2009-01-27: (permalink)
"Did people experience this on Hardy (8.04) as well, or just in Intrepid (8.10)?"

yes, Hardy also! (Ubuntu 8.04.2 )

When approximatelly could the patch be also available for it via "get-apt update/upgrade" ?
Valery

Revision history for this message
LumpyCustard (orangelumpycustard) wrote :

I just installed this update and opted to 'Override' the settings (at least I think that was the word).

Now I have no sudo access. My password has been flattened. Any ideas?

Revision history for this message
Martin Pitt (pitti) wrote : Re: [Bug 284229] Re: console-kit-daemon using a lot of cpu

Hello,

LumpyCustard [2009-02-16 12:48 -0000]:
> I just installed this update and opted to 'Override' the settings (at
> least I think that was the word).

What did that dialog say exactly? What "settings" do you mean?

> Now I have no sudo access. My password has been flattened. Any ideas?

No, not really. ConsoleKit does not ever touch passwords. Can you
start rescue mode, do a root shell, use

  id <yourloginname>

to check whether you are still in the "admin" group, and if so, do

  EDITOR=nano visudo

and check whether you have

  %admin ALL=(ALL) ALL

?

Revision history for this message
LumpyCustard (orangelumpycustard) wrote :

Martin,

the prompt occurred whilst installing the new libpam-ck-connector.

admin ALL=(ALL) ALL was still there, and I checked and I was in the admin group. I'm still not sure what happened, but it doesn't matter now. I ended up formatting and reinstalling everything (which I was meaning to do anyway to use ext4). All this did was speed things up.

You were right when you said my password was still active, but it just didn't seem to want to be accepted as the sudo password.

I tried to replicate the update in a VM but couldn't. I have another machine exactly the same as mine which I am going to update. If I see the same prompt, I'll post back here what was said.

Revision history for this message
vak (khamenya) wrote :

hi all

any cure for Hardy(Ubuntu 8.04.2 ) already?..

regards
Valery

Revision history for this message
vak (khamenya) wrote :

i've checked output of "apt-rdepends -r consolekit". It is huge... uninstall consolekit is a no-go. Without a patch Hardy(Ubuntu 8.04.2) becomes hardly usable... :-/

any news?

Revision history for this message
Nathan Crawford (njcrawford) wrote :

@vak - I'm not able to reproduce this on 8.04.2 using the steps to reproduce above. Do you have a different method to reproduce it on Hardy?

Revision history for this message
Samuel Kleiner (cultureulterior) wrote :

This bug also affects Jaunty.

With 0.3.0-2ubuntu3 i see

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3323 root 20 0 2234m 1.7g 1576 R 64 22.6 2429:41 console-kit-dae

Changed in consolekit:
importance: Unknown → High
Changed in consolekit:
importance: High → Unknown
Changed in consolekit:
importance: Unknown → High
Revision history for this message
Whit Blauvelt (whit-launchpad) wrote :

Just had a serious runaway console-kit-daemon on Lucid! Ran my load up into the 40-50 range with 100% CPU. Killall console-kit-daemon reduced load to under 0.30. Is this a regression, or a newer bug?

Revision history for this message
Brenton P (bapartridge) wrote :

No runaway CPU, but I just had ~60 console-kit-daemons leaking after a day's worth of frequent Capistrano deploys (which launch multiple SSH sessions) on a brand-new EC2 instance running Ubuntu 12.10. Probably not your issue, Whit, but the leakiness is still concerning.

Revision history for this message
Stack Underflow (fathi) wrote :

consile-kit-dae is not eating CPU on my system right now but it is taking 4GB of memory.

$ top
2609 root 20 0 4090m 1772 1484 S 0 0.0 0:10.39 console-kit-dae

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.