console-kit-daemon crashed with SIGSEGV in vt_thread_start()

[Andre]

LatestPackageTested: consolekit 0.2.3-3ubuntu5

Binary package hint: consolekit

I don't know how to reproduce this but after reboot it appeared.

ProblemType: Crash
Architecture: i386
Date: Sat Jan 19 22:34:09 2008
DistroRelease: Ubuntu 8.04
ExecutablePath: /usr/sbin/console-kit-daemon
NonfreeKernelModules: ieee1394
Package: consolekit 0.2.3-3ubuntu2
PackageArchitecture: i386
ProcCmdline: /usr/sbin/console-kit-daemon
ProcCwd: /
ProcEnviron:

Signal: 11
SourcePackage: consolekit
Stacktrace:
 #0 0x08050399 in ?? ()
 #1 0x00000000 in ?? ()
StacktraceTop:
 ?? ()
 ?? ()
Title: console-kit-daemon crashed with SIGSEGV
Uname: Linux snoopy 2.6.24-4-generic #1 SMP Mon Jan 14 17:30:39 UTC 2008 i686 GNU/Linux
UserGroups:

[Apport retracing service (apport)]

StacktraceTop:vt_thread_start (data=0x806a7a0) at ck-vt-monitor.c:289
g_thread_create_proxy (data=0x8069898) at /build/buildd/glib2.0-2.15.3/glib/gthread.c:628
start_thread () from /lib/tls/i686/cmov/libpthread.so.0
clone () from /lib/tls/i686/cmov/libc.so.6

[Remove Me (remove-me)]

From Ubuntu: https://bugs.launchpad.net/bugs/148454

since upgrading to gutsy beta htop shows about fifty processes like the following

pid user pri ni virt res shr s cpu% mem% time+ command
7582 root 23 0 7456 2016 1336 s 0.0 0.2 0:00.00 /usr/sbin/console-kit-deamon

It maybe the case that console-kit-daemon starts a thread for each console that can theoretically exist. On Linux (/usr/include/linux/vt.h) MAX_NR_CONSOLE is defined to be 63.

Note that those processes do not show up in DEFAULT top or ps output --BUT-- if you run either of those tools with the -H parameter, so as to display all the threads of a process, then they do show up.

[Martin Pitt (pitti)]

This might also explain https://launchpad.net/bugs/184519, which has 26 duplicates already, with this stack trace:

  http://launchpadlibrarian.net/12890099/Stacktrace.txt

(crash in vt_thread_start). Or was that actually fixed by dfcab49480565a7bcf71752c5b39eb367df81a19 "cleanly shutdown event logging threads"?

[Martin Pitt (pitti)]

Ah, to answer myself, this is probably the case: https://launchpad.net/bugs/196724 is another set of 50 duplicates about the same "crash in thread start" problem, which was fixed with the 0.3 patch I mentioned.

[Martin Pitt (pitti)]

This has recently been fixed in Intrepid, see bug 196724. I will use that one for the hardy stable update.

[Linus Walleij (triad)]

I experience these 63 spawned processes as well, if for nothing else
(all CoW memory etc accounted for) it hogs the scheduler by making
it slower to sort the CFS RB-tree which is O(log n).

An Ubuntu user proposed the following patch to
autoconfigure the maximum number of consoles from sysfs:

http://launchpadlibrarian.net/15940846/0001-Autoconfigure-maximum-number-of-kernel-consoles.patch

which looks sane to me.

I made a graphical tool to display process migration etc in the system
and the plethora of console-kit-daemons really cloud the view.

[Remove Me (remove-me)]

(In reply to comment #3)
> I experience these 63 spawned processes as well, if for nothing else
> (all CoW memory etc accounted for) it hogs the scheduler by making
> it slower to sort the CFS RB-tree which is O(log n).
>
> An Ubuntu user proposed the following patch to
> autoconfigure the maximum number of consoles from sysfs:
>
> http://launchpadlibrarian.net/15940846/0001-Autoconfigure-maximum-number-of-kernel-consoles.patch
>
> which looks sane to me.
>
> I made a graphical tool to display process migration etc in the system
> and the plethora of console-kit-daemons really cloud the view.
>

I am that user. The patch broke the logout process of Ubuntu: the
currently logged in user lost the ability to power off the system,
only logout worked.

[William Jon McCann (william-jon-mccann)]

Sorry for the long delay. However, it seems like this bug has a few different issues in it. The issue that is reported in the initial report is actually not a bug but a design decision. Once we improve or replace the VT subsystem we can do better than this. Until then we don't have another way.

[Linus Walleij (triad)]

Aha it's not a bug it's a feature.

[micah4 (atomicsuntan)]

Created an attachment (id=24210)
Patch to console kit to use alternative wait mechanism

Here is a patch for console kit that removes the numerous threads by utilizing a new kernel ioctl. (This requires a kernel patch as well).

[micah4 (atomicsuntan)]

Created an attachment (id=24211)
Kernel patch adds a VT_WAITSWITCH ioctl command

This patch adds an ioctl to the kernel which console-kit can use to wait on a terminal switch without creating 60 threads. It's not incredibly pretty but it seems to work and demonstrates a straightforward solution.

[micah4 (atomicsuntan)]

Does anybody know who the official maintainer of this package is? I am willing to work on getting the patch included in the kernel if this approach is acceptable to the ConsoleKit team, but I don't want to bother with the kernel side of things unless ConsoleKit will incorporate the changes also. I tried emailing Jon McCann but did not get any response.

[Michael Biebl (mbiebl)]

(In reply to comment #9)
> Does anybody know who the official maintainer of this package is? I am willing

William Jon McCann is the official maintainer.

> to work on getting the patch included in the kernel if this approach is
> acceptable to the ConsoleKit team, but I don't want to bother with the kernel
> side of things unless ConsoleKit will incorporate the changes also. I tried
> emailing Jon McCann but did not get any response.
>

That's very unfortunate. I can only assume he is very busy. Maybe it helps to ping him again.

[Matthias Clasen (mclasen)]

> Does anybody know who the official maintainer of this package is? I am willing
> to work on getting the patch included in the kernel if this approach is
> acceptable to the ConsoleKit team, but I don't want to bother with the kernel
> side of things unless ConsoleKit will incorporate the changes also. I tried
> emailing Jon McCann but did not get any response.

This kind of hedging typically doesn't work were well in free software projects.
Don't let the lack of preapproval here keep you from getting the kernel side of things fixed.

[Lennart-poettering (lennart-poettering)]

BTW:

http://lkml.indiana.edu/hypermail/linux/kernel/0906.3/02726.html

Alan wanted cook up his own patch, but uh, he didn't respond in a week or so.

[William Jon McCann (william-jon-mccann)]

Lennart, was there any followup from Alan on this?

[Lennart-poettering (lennart-poettering)]

(In reply to comment #13)
> Lennart, was there any followup from Alan on this?
>

Alan gave up maintainership of the console stuff a few weeks back. gregkh then took this up and one of the first things he did is merge Alan's patch into his tree. Two days ago this was then merged into mainline. So I guess that means the kernel infrstructure should now be able to cut down the threads to two (the ioctl is still blocking)

[William Jon McCann (william-jon-mccann)]

Lennart, any chance you can make a patch to use this new functionality?

[Rez (rez)]

This 3+ years old bug (or feature...) is still giving many people headaches: https://bugs.launchpad.net/ubuntu/+source/consolekit/+bug/148454

Any chance to see those useless processes go away in our lifetime?

If Alan gave up maintainership can someone please work on a fixed version of that ck autoconfigure patch, or something different, an hack, anything?

[Lennart-poettering (lennart-poettering)]

(In reply to comment #16)
> Any chance to see those useless processes go away in our lifetime?

They aren't processes, they are threads. And they serve a purpose.

The current behaviour might be suboptimal, but it is correct, as such it would be nice if this is fixed but we certainly have more burning issues to fix.

Also note that whatever the LP bug says about the 63 threads wasting oh so much resources, that is simply nonsense, the folks complaining have no clue what they are speaking of.

> If Alan gave up maintainership can someone please work on a fixed version of
> that ck autoconfigure patch, or something different, an hack, anything?

The stuff is in the kernel. It's just a matter of someone writing a patch for CK. Would be wonderful if you and your Ubuntu friends might contribute good code once in a while instead of just complaining for 3 years straight... ;-)

Anyway, I might come up with a patch eventually, but it is not a priority item on my todo list, sorry.

[Rez (rez)]

(In reply to comment #17)

Maybe we're complaining because we're users, not developers. I didn't *choose* to install consolekit in the first place... but I have to, or stay away from X/Gnome/nm and who knows what else.

Anyway, several console-kit/kernel patches *have* been proposed on LP by those "ubuntu friends" you were talking about; unfortunately, limiting consolekit seems to cause other problems. That's why most people are just waiting it to be fixed in upstream.

[Lennart-poettering (lennart-poettering)]

(In reply to comment #18)
> Anyway, several console-kit/kernel patches *have* been proposed on LP by those
> "ubuntu friends" you were talking about; unfortunately, limiting consolekit
> seems to cause other problems. That's why most people are just waiting it to be
> fixed in upstream.

if the patches are good, why aren't they submitted upstream?

[rifter (rifter0x0000)]

(In reply to comment #17)
> (In reply to comment #16)
> > Any chance to see those useless processes go away in our lifetime?
>
> They aren't processes, they are threads. And they serve a purpose.
>
> The current behaviour might be suboptimal, but it is correct, as such it would
> be nice if this is fixed but we certainly have more burning issues to fix.
>
> Also note that whatever the LP bug says about the 63 threads wasting oh so much
> resources, that is simply nonsense, the folks complaining have no clue what
> they are speaking of.
>
> > If Alan gave up maintainership can someone please work on a fixed version of
> > that ck autoconfigure patch, or something different, an hack, anything?
>
> The stuff is in the kernel. It's just a matter of someone writing a patch for
> CK. Would be wonderful if you and your Ubuntu friends might contribute good
> code once in a while instead of just complaining for 3 years straight... ;-)
>
> Anyway, I might come up with a patch eventually, but it is not a priority item
> on my todo list, sorry.

I have to disagree with the claim that resources are not being eaten up by console-kit-daemon. According to a top I did on a system that was recently having memory issues, the 64 console kit processes were using over 7.6GB of memory (122+MB each). as someone else has pointed out, patches have been proposed for this. Maybe they were not "good code," I haven't read it. It would be interesting to find out why there need to be so many processes at outset rather that spawning a more conservative number of processes (or threads as the case may be; although each one seemed to have a distinct pid) to me even using 122MB total would be silly; what is this program doing that requires so much RAM?

[aquo (sbauch)]

Created an attachment (id=36072)
Patch to make maximal number of watched VTs configurable

I once wrote a small patch for an older ubuntu version that makes it possible to limit the number of watched VTs. The patch is old, for consolekit-0.2.3 and I don't know if it still applies. The main idea was to make the maximal number of threads configurable to the user.

https://bugs.launchpad.net/consolekit/+bug/148454/comments/35

I would like to know if it makes sense to watch Virtual Terminals that have no gettys attached?

[Lennart-poettering (lennart-poettering)]

A few kernel releases back a new kernel API was added that makes the many threads unnecessary. Instead of coming up with ugly work-arounds for the current "problem", please fix things properly and just prepare a patch that ports CK to the new API entirely.

[Kanru Chen (kanru)]

Created an attachment (id=37970)
New patch that uses new VT_WAITEVENT ioctl

New patch that uses new VT_WAITEVENT ioctl.

[Lennart-poettering (lennart-poettering)]

I commited a slightly modified version of Kan-Ru's patch now to git. Thanks!

[Linus Walleij (triad)]

Thanks for fixing this Lennart!

[Lennart-poettering (lennart-poettering)]

(In reply to comment #25)
> Thanks for fixing this Lennart!

Say thanks to Kan-Ru, he did the patch!