console-kit-daemon leaks memory

[radsaq]

Binary package hint: consolekit

After 23 days of uptime, the console-kit-daemon process is using up 168M of memory:

root 5462 0.0 8.3 213208 172740 ? Ssl Apr28 3:53 /usr/sbin/console-kit-daemon

ProblemType: Bug
Architecture: i386
Date: Wed May 21 12:09:35 2008
DistroRelease: Ubuntu 8.04
ExecutablePath: /usr/sbin/console-kit-daemon
NonfreeKernelModules: nvidia
Package: consolekit 0.2.3-3ubuntu5
PackageArchitecture: i386
ProcEnviron:

SourcePackage: consolekit
Uname: Linux 2.6.24-16-generic i686

[Colinjones71 (colinjones71)]

I'm using Ubuntu 9.10 (consolekit - 0.3.1-0ubuntu2) and have noticed that this daemon continually sucks up more and more physical memory over time, even when the machine is not being used.

It starts around 3.7MB, but then increases. After about 1-2 days it reaches 200MB, I presume it would continue indefinitely but I usually reboot at that point.

I am also trying to chase down another memory issue that massively impacts performance. The box has 2GB of RAM, but something somewhere seems to be using it, but it isn't accounted for anywhere. free -m indicates that it has been used (ignoring cache and buffers of course!), as does System Monitor. But adding up all the processes' Memory (in S.M.) or all the RES/RSS in ps or top falls many hundreds of MBs short, and this gap increases. Quite quickly it gets to the point where the box is swapping constantly, when the number/type/use of apps shouldn't require anything like it. I do not have any reason to assume this is related, but mention it just in case.

Last night, not long after a reboot and before going to bed: (10MB)
root 891 1.0 0.4 122652 9840 ? Ssl 21:38 0:23 /usr/sbin/console-kit-daemon

This morning after box being unused all last night: (100MB)
root 891 2.2 5.2 221388 108180 ? Ssl Jan25 13:33 /usr/sbin/console-kit-daemon

[Colinjones71 (colinjones71)]

pmap seems to indicate something weird... libraries are being mapped with 2MB chunks each with no permissions against them.

colin@iluvatar:~$ sudo pmap -d 891
891: /usr/sbin/console-kit-daemon
Address Kbytes Mode Offset Device Mapping
0000000000400000 128 r-x-- 0000000000000000 008:00001 console-kit-daemon
000000000061f000 4 r---- 000000000001f000 008:00001 console-kit-daemon
0000000000620000 4 rw--- 0000000000020000 008:00001 console-kit-daemon
0000000001af3000 36980 rw--- 0000000000000000 000:00000 [ anon ]
00007f4778000000 33188 rw--- 0000000000000000 000:00000 [ anon ]
00007f477a069000 32348 ----- 0000000000000000 000:00000 [ anon ]
00007f4780000000 38020 rw--- 0000000000000000 000:00000 [ anon ]
00007f4782521000 27516 ----- 0000000000000000 000:00000 [ anon ]
00007f4785ead000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4785eae000 10240 rw--- 0000000000000000 000:00000 [ anon ]
00007f47868ae000 88 r-x-- 0000000000000000 008:00001 libgcc_s.so.1
00007f47868c4000 2044 ----- 0000000000016000 008:00001 libgcc_s.so.1
00007f4786ac3000 4 r---- 0000000000015000 008:00001 libgcc_s.so.1
00007f4786ac4000 4 rw--- 0000000000016000 008:00001 libgcc_s.so.1
00007f4786ade000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786adf000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786aef000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786af0000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b00000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b01000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b11000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b12000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b22000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b23000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b33000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b34000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b44000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b45000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b55000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b56000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b66000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b67000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b77000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b78000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b88000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b89000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786b99000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786b9a000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786baa000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786bab000 64 rw--- 0000000000000000 000:00000 [ anon ]
00007f4786bbb000 4 ----- 0000000000000000 000:00000 [ anon ]
00007f4786bbc000 64 rw--- 0000000000000000 000:00000 [ anon ]
0000...

[William Jon McCann (william-jon-mccann)]

Can you reproduce this with the latest release or from git? Thanks. Might be good to try to get some valgrind info too.

[Colinjones71 (colinjones71)]

(In reply to comment #2)
> Can you reproduce this with the latest release or from git? Thanks. Might be
> good to try to get some valgrind info too.

eek, a bit beyond me! I don't suppose there is a deb anywhere that I can use on Ubuntu 0910 64bit? Compiling from source is only going to get me in trouble!

[Colinjones71 (colinjones71)]

Hello?? I have downloaded the latest source, but can't even work out how to compile it and there are no instructions. Doesn't seem to be the usual ./configure ,,, make ... make install process....

[Colinjones71 (colinjones71)]

seriously, I need help here! After only 1 day and 6 hours uptime:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
  959 root 20 0 405m 291m 1864 S 13 14.5 82:21.37 console-kit-dae
18974 colin 20 0 751m 148m 13m S 1 7.4 16:25.05 firefox
 2242 root 20 0 199m 141m 2160 S 1 7.0 7:33.82 polkitd
27313 colin 20 0 784m 121m 14m S 0 6.1 2:08.98 cairo-dock
16807 root 20 0 290m 113m 12m S 3 5.6 26:06.31 Xorg
17198 colin 20 0 359m 86m 5608 S 0 4.3 2:55.99 polkit-gnome-au

Console-kit-daemon, polkitd and polkit-gnome-authentication-agent-1 are consuming over 0.5GB between them!

I would love to try the latest version, but simply can't work out how to compile it, what compilation tools I need, and what other dependencies there are...

please can you help, as this is crippling my machine daily.

[Olafur Arason (olafura)]

For me it's virtual memory that it's eating and runs up to ~350mb. That does not work well in my system and it does this repeatedly.

[Olafur Arason (olafura)]

I'm using Karmic on x86.

[Colinjones71 (colinjones71)]

Forget it.... its perfectly clear from all your other tickets that this is your modus operandi, its a shame such a great project has such an unresponsive, discourtious dev involved.

[Freedesktop-mikeanddanelle (freedesktop-mikeanddanelle)]

I'm running Ubuntu 9.10 x64 with consolekit 0.3.1-0ubuntu2. This workstation has been setting idle fresh from a reboot without logging in for the past two days, and I appear to be suffering from the same issue as this bug report describes.

 18:21:05 up 2 days, 2:54, 4 users, load average: 1.07, 1.52, 1.75

Mem: 2056992k total, 1989884k used, 67108k free, 94448k buffers
Swap: 6024332k total, 78180k used, 5946152k free, 300644k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1192 root 20 0 753m 558m 1932 S 10 27.8 248:28.92 console-kit-dae

My pmap looks very similar, only larger numbers.

I am familiar with valgrind, but I haven't dug into how consolekit works. If someone could give me some quick guidance on how to profile consolekit. I'll have 20 or 30 minutes tomorrow evening to try to dig into why this is happening.

[BlueWolf (josr91)]

This is a really bad bug for my 9.04 (32bit) server. It has 244MB memory and 345MB of swap. Everytime after about a month or 2 my server starts crashing and becomes unavailable from everywhere. A hard reset is the only thing that helps. Turns out it was out of memory and it started killing random daemons. I just found out that console-kit-daemon is using 20% of the available memory. After I killed the console-kit-daemon, *355MB* of memory came free, almost as big as my whole swap. A workaround would be really nice.

[Anders Kaseorg (andersk)]

At least one memory leak is easy to reproduce by running a loop like

  ssh-copy-id localhost
  while true; do ssh localhost true; done

and watching console-kit-daemon’s resident memory usage go up by tens of kilobytes per second in `top`.

(I’m running consolekit 0.4.1-4 on Ubuntu maverick amd64.)

[Anders Kaseorg (andersk)]

Created an attachment (id=37533)
patches for a few bad memory leaks

These three patches fix most of the really bad (per-session) memory leaks I found with valgrind. (Apply with `git am`.)

There is still one worrisome leak within dbus-glib that might or might not be ConsoleKit’s fault:

==4766== 2,400 bytes in 150 blocks are definitely lost in loss record 997 of 1,005
==4766== at 0x4C284A8: malloc (vg_replace_malloc.c:236)
==4766== by 0x6282514: g_malloc (gmem.c:134)
==4766== by 0x6297749: g_slice_alloc (gslice.c:836)
==4766== by 0x6298F62: g_slist_append (gslist.c:229)
==4766== by 0x4E394C1: dbus_g_connection_register_g_object (dbus-gobject.c:2373)
==4766== by 0x40F127: register_session (ck-session.c:147)
==4766== by 0x40F577: ck_session_new_with_parameters (ck-session.c:1249)
==4766== by 0x408FC7: collect_parameters_cb (ck-manager.c:1621)
==4766== by 0x40DF4F: job_completed (ck-session-leader.c:390)
==4766== by 0x59EB69D: g_closure_invoke (gclosure.c:766)
==4766== by 0x5A02DA8: signal_emit_unlocked_R (gsignal.c:3252)
==4766== by 0x5A04525: g_signal_emit_valist (gsignal.c:2983)

But at least it’s much smaller than the ones I fixed.

[William Jon McCann (william-jon-mccann)]

Review of attachment 37533:

Thanks for tracking these down! The review tool doesn't cope very well with having two different patches for the same file apparently so I'm not sure this review will look as intended. Apart from the unref issues I mentioned this looks good.

::: src/ck-manager.c
@@ +1256,1 @@
 }

This isn't right. Functions shouldn't generally unref passed in parameters as a side effect.

@@ +1273,1 @@
 }

Also not right for the same reason.

@@ +1287,3 @@
         log_seat_session_removed_event (manager, seat, ssid);
+
+ g_free (ssid);

We don't need to take a new ref here since we already have one in this context.

[Anders Kaseorg (andersk)]

> This isn't right. Functions shouldn't generally unref passed in parameters as
> a side effect.

I’m pretty sure I needed to do it that way, because ck_session_leader_collect_parameters stashes the callback and the leader away in the pending_jobs queue rather than running the callback immediately. With your modified version of my patch that you committed, is there anything to prevent the leader from being freed while the job is still in the queue?

[William Jon McCann (william-jon-mccann)]

(In reply to comment #11)
> your modified version of my patch that you committed, is there anything to
> prevent the leader from being freed while the job is still in the queue?

The hash table holds a ref. So I don't think it should happen.

[Anders Kaseorg (andersk)]

(In reply to comment #12)
> The hash table holds a ref. So I don't think it should happen.

But still, what stops the hash table entry from being removed while the job is still in the queue?

[Jan K. (jan-launchpad-kantert)]

Hi,

we see the same issue in Ubuntu 10.04.1 LTS on multiple maschines. After a few days it ramps up to about 25% of System RAM (Virt: 626MB Res: 473MB).

Regards,
Jan

[W.D. (wdr)]

Just to inform that I have the problem under Ubuntu 8.04 (desktop).
After 11 days of uptime ps axu gives
root 6062 0.0 4.7 107924 98684 ? Ssl Sep15 2:38 /usr/sbin/console-kit-daemon

Moreover, the only cure that works for me is reboot (after having killed console-kit-daemon I cannot get it back again).

Notice that the rather confused discussion in https://bugs.launchpad.net/ubuntu/+source/consolekit/+bug/148454 contains quite a few examples of the same memory leak problem.

Maybe at least a hint how to restart the program?

best regards
--W.D.

[Matthias Urlichs (smurf)]

Quite frankly, the patch look like it makes a whole lot of sense.

Meanwhile, users see their servers die or hang because the daemon eats memory like crazy. Please either apply the patch, or find some other way to stop the leak. Thank you.

[Matthias Urlichs (smurf)]

*sigh* a machine-crippling memory leak bug, open for nine months.

The Upstream bug has a patch. It may not be perfect, but please consider applying it. The current memory leak definitely needs to be fixed.

[Anders Kaseorg (andersk)]

(In reply to comment #14)
> Please either apply the patch, or find some other way to stop the leak.

William already applied a modified version of my patch for 0.4.2. The leak should already be gone.
http://cgit.freedesktop.org/ConsoleKit/commit/?id=7b9212fa6aff55420c58f2cacd0a941762920337

In the discussion that followed, I was just worried that his modifications introduced a potential use-after-free crash, because the JobData created by ck_session_leader_collect_parameters isn’t holding a reference to its leader. If there is no way for a CkSessionLeader to be freed with entries remaining in its pending_jobs queue (previously this was prevented by the missing unref in create_session_for_sender), then that isn’t a concern and this can be closed. Otherwise, perhaps this presentation makes it clearer why the extra ref and unref are okay?

--- a/src/ck-session-leader.c
+++ b/src/ck-session-leader.c
@@ -409,6 +409,7 @@ job_completed (CkJob *job,
 static void
 job_data_free (JobData *data)
 {
+ g_object_unref (data->leader);
         g_free (data);
 }

@@ -428,7 +429,7 @@ ck_session_leader_collect_parameters (CkSessionLeader *session_leader,
         ret = FALSE;

         data = g_new0 (JobData, 1);
- data->leader = session_leader;
+ data->leader = g_object_ref (session_leader);
         data->done_cb = done_cb;
         data->user_data = user_data;
         data->context = context;

[Anders Kaseorg (andersk)]

This should be fixed in ConsoleKit 0.4.2, which is now in natty:

consolekit (0.4.2-1ubuntu1) natty; urgency=low

  * Resynchronise with Debian. Remaining changes:
    - 10-retry_console_open_eio.patch: Retry console opens if they return
      EIO, since this may happen while a tty is closing.

 -- Colin Watson <email address hidden> Wed, 13 Oct 2010 12:28:33 +0100

[Jan K. (jan-launchpad-kantert)]

As a workaround we simply removed console-kit-daemon from our servers. Seems not to be needed for servers. Resolved all our problems so far!

[BlueWolf (josr91)]

I started to use Munin to graph the health of my server (9.04). A few days ago I killed console-kit-daemon (which is now a regular task for me), and I just saw these graphs. Apart from the huge memory drop, I also noticed a huge processes drop at the same time I killed this daemon. From 200 to 100 processes! Did killing console-kit-daemon really killed hundred processes?

It's nice it's fixed in Natty, but I'm not using Natty. Can you safely remove this console-kit-daemon? If so, how? By just removing/rename the executable file? I want to trust my server to keep running even when I'm not continuously monitoring it.

[W.D. (wdr)]

Message #22 says: status: Confirmed -> Fix Released
But how to apply the fix to older versions of Ubuntu?
Is it possible to install consolekit (0.4.2-1ubuntu1) natty on those versions?
If so, how?

Also the questions of #20 and #24 remain unanswered.

I have an addition to what I wrote in #20.
I am using Ubuntu 8.04 LTS with Gnome replaced by KDE, on a Thinkpad laptop.
Under my 8.04 the problem seems to disappear when kernel 2.6.31 is used
(basically from Ubuntu 9.10):

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4007 root 20 0 9060 1704 1372 S 0 0.1 0:00.22 console-kit-dae

This is the situation under the original kernel 2.6.24, after 28 days
of uptime:

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
6083 root 20 0 202m 194m 1608 S 0 9.7 5:35.92 console-kit-dae

(Sorry, the lines of 'top' display ugly.)

[John Dong (jdong)]

I'd say since the commit http://cgit.freedesktop.org/ConsoleKit/commit/?id=7b9212fa6aff55420c58f2cacd0a941762920337 has been accepted in upstream git and is in Natty, we should consider SRU'ing this to Lucid and Maverick at the very least. It is a potentially high-impact bug. I've come across one server in which console-kit-daemon was using 6GB of RAM.

[Anders Kaseorg (andersk)]

(In reply to comment #15)
> + g_object_unref (data->leader);
> …
> - data->leader = session_leader;
> + data->leader = g_object_ref (session_leader);

William, do you have any comments on this follow-up patch (comment #15)? Or even just a reason that a CkSessionLeader can’t be freed with entries remaining in its pending_jobs queue, and hence the patch wouldn’t be necessary?

[W.D. (wdr)]

On 2011-01-27 I wrote:
> Under my 8.04 the problem seems to disappear when kernel 2.6.31 is used

I cannot reproduce this anymore. The memory leak is still there under this kernel too.

[Harald (haraldboehmecke)]

I see this bug hasn't been worked on for a while. Issue is still present!

root@ccs:/home/milz04# pmap -d 1452
1452: /usr/sbin/console-kit-daemon --no-daemon
Address Kbytes Mode Offset Device Mapping
08048000 116 r-x-- 0000000000000000 008:00001 console-kit-daemon
08065000 4 r---- 000000000001c000 008:00001 console-kit-daemon
08066000 4 rw--- 000000000001d000 008:00001 console-kit-daemon
08892000 981868 rw--- 0000000000000000 000:00000 [ anon ]
b6500000 132 rw--- 0000000000000000 000:00000 [ anon ]
b6521000 892 ----- 0000000000000000 000:00000 [ anon ]
b663b000 28 r--s- 0000000000000000 008:00001 gconv-modules.cache
b6642000 4 ----- 0000000000000000 000:00000 [ anon ]
b6643000 10240 rw--- 0000000000000000 000:00000 [ anon ]
b7043000 4 ----- 0000000000000000 000:00000 [ anon ]
b7044000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7054000 4 ----- 0000000000000000 000:00000 [ anon ]
b7055000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7065000 4 ----- 0000000000000000 000:00000 [ anon ]
b7066000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7076000 4 ----- 0000000000000000 000:00000 [ anon ]
b7077000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7087000 4 ----- 0000000000000000 000:00000 [ anon ]
b7088000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7098000 4 ----- 0000000000000000 000:00000 [ anon ]
b7099000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70a9000 4 ----- 0000000000000000 000:00000 [ anon ]
b70aa000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70ba000 4 ----- 0000000000000000 000:00000 [ anon ]
b70bb000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70cb000 4 ----- 0000000000000000 000:00000 [ anon ]
b70cc000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70dc000 4 ----- 0000000000000000 000:00000 [ anon ]
b70dd000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70ed000 4 ----- 0000000000000000 000:00000 [ anon ]
b70ee000 64 rw--- 0000000000000000 000:00000 [ anon ]
b70fe000 4 ----- 0000000000000000 000:00000 [ anon ]
b70ff000 64 rw--- 0000000000000000 000:00000 [ anon ]
b710f000 4 ----- 0000000000000000 000:00000 [ anon ]
b7110000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7120000 4 ----- 0000000000000000 000:00000 [ anon ]
b7121000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7131000 4 ----- 0000000000000000 000:00000 [ anon ]
b7132000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7142000 4 ----- 0000000000000000 000:00000 [ anon ]
b7143000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7153000 4 ----- 0000000000000000 000:00000 [ anon ]
b7154000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7164000 4 ----- 0000000000000000 000:00000 [ anon ]
b7165000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7175000 4 ----- 0000000000000000 000:00000 [ anon ]
b7176000 64 rw--- 0000000000000000 000:00000 [ anon ]
b7186000 4 ----- 0000000000000000 000:00000 [ anon ]
b7187000 64 rw--...

[Scott Ritchie (scottritchie)]

I have ran into this too, and it's really messing up my servers. I'm going to put up a backport of the Precise version in a PPA.

[Scott Ritchie (scottritchie)]

I've been using my PPA version on a few thousand ubuntu server cloud nodes over the past month and the problem has disappeared. https://launchpad.net/~scottritchie/+archive/consolekit-backports/+packages

I uploaded it to lucid-proposed, awaiting SRU.

[Scott Ritchie (scottritchie)]

Turns out I don't have main rights and can't upload this to -proposed, so have a debdiff and please sponsor :)

[Jan Evert van Grootheest (j-e-van-grootheest)]

It is insane. One user logged in and out. And 2G of memory is data-resident?

  PID TTY STAT TIME MAJFL TRS DRS RSS %MEM COMMAND
 2155 ? Sl 0:00 1 0 2091744 4012 0.1 /usr/sbin/console-kit-daemon --no-daemon