console-kit-daemon Glib-CRITICAL warning

[Ariel Faigon]

The day I upgraded from 13.10 to 14.04 some of my ssh sessions into the new system (server side is Ubuntu 14.04) started hanging.

The symptom is: I run some command with a lot of output (say a few hundreds of lines) in an ssh session, and the second the command completes, and the shell displays a new prompt, the shell stops responding to keyboard input.

At the exact time of the hang (to the second) I always see these 3 warnings in syslog:

Apr 21 18:15:25 xx console-kit-daemon[3357]: GLib-CRITICAL: Source ID 469 was not found when attempting to remove it
Apr 21 18:15:25 xx console-kit-daemon[3357]: GLib-CRITICAL: Source ID 86 was not found when attempting to remove it
Apr 21 18:15:25 xx console-kit-daemon[3357]: GLib-CRITICAL: Source ID 86 was not found when attempting to remove it

The Source ID numbers vary, but the warnings always come in triplets at the time of the hang. I was able to reproduce this about 10 times today. Every time the session hangs, these warnings appear in the log and vice versa: every time they appear in the log, I have a hanged remote session.

A quote from an upstream (bugzilla.gnome.org) comment:

https://bugzilla.gnome.org/show_bug.cgi?id=721369

    GLib recently started throwing a warning when g_source_remove()
    is passed garbage (as per warning). Your applications have probably been broken
    for a while, and there's no telling what could actually have happened in the
    past when g_source_remove() would happily close any random source because the
    programmer got the wrong argument to g_source_remove().

So based on this I'm opening a bug against consolekit for bad calls to g_source_remove() which seem to be related to the ssh hang.

[Ariel Faigon (ariel.faigon)]

Asked two other users to ssh remotely to my server and noted that the GLib error can also appear on a normal session close (exit the shell) . This is more evidence that the Glib-CRITICAL error is a symptom rather than the cause of the ssh-hang bug. It is also evidence that both successful and buggy-hang session-closes are all resulting in the GLib-CRITICAL error.

In the (no hang) cases, I see only two GLib-CRITICAL errors, as opposed to 3.

Apr 26 02:24:43 go sshd[29233]: pam_unix(sshd:session): session opened for user zzzz by (uid=0)
Apr 26 02:24:43 go systemd-logind[607]: New session 143 of user zzzz.
Apr 26 02:24:43 go systemd-logind[607]: Removed session 142.
[...session without issues and no hang until exit happens here...]
Apr 26 02:25:15 go console-kit-daemon[3357]: GLib-CRITICAL: Source ID 4207 was not found when attempting to remove it
Apr 26 02:25:15 go sshd[29233]: pam_unix(sshd:session): session closed for user zzzz
Apr 26 02:25:15 go sshd[29285]: Received disconnect from w.x.y.z: disconnected by user

[Guido Maria Serra (zeph1ro)]

hey, I don't think this is console-kit specific, but relates only to GLib

I'm getting the same error from Sublime Text ....

(sublime_text:12832): GLib-CRITICAL **: Source ID 239376 was not found when attempting to remove it

(sublime_text:12832): GLib-CRITICAL **: Source ID 242857 was not found when attempting to remove it

(sublime_text:12832): GLib-CRITICAL **: Source ID 244503 was not found when attempting to remove it

[Guido Maria Serra (zeph1ro)]

$ file sublime_text
sublime_text: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, BuildID[sha1]=17ad7e4cfbc3c3c7919f99584a922b7c8ceea690, stripped

$ ldd sublime_text
 linux-vdso.so.1 => (0x00007fff3c2d2000)
 libgobject-2.0.so.0 => /usr/lib/x86_64-linux-gnu/libgobject-2.0.so.0 (0x00007f44e654f000)
 librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f44e6347000)
 libglib-2.0.so.0 => /lib/x86_64-linux-gnu/libglib-2.0.so.0 (0x00007f44e603e000)
 libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f44e5e20000)
 libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f44e5c1c000)
 libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f44e5a18000)
 libX11.so.6 => /usr/lib/x86_64-linux-gnu/libX11.so.6 (0x00007f44e56e3000)
 libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f44e53df000)
 libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f44e50d8000)
 libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f44e4ec2000)
 libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f44e4afc000)
 libffi.so.6 => /usr/lib/x86_64-linux-gnu/libffi.so.6 (0x00007f44e48f3000)
 libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f44e46b5000)
 /lib64/ld-linux-x86-64.so.2 (0x00007f44e67c5000)
 libxcb.so.1 => /usr/lib/x86_64-linux-gnu/libxcb.so.1 (0x00007f44e4495000)
 libXau.so.6 => /usr/lib/x86_64-linux-gnu/libXau.so.6 (0x00007f44e4291000)
 libXdmcp.so.6 => /usr/lib/x86_64-linux-gnu/libXdmcp.so.6 (0x00007f44e408b000)

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty

[Ariel Faigon (ariel.faigon)]

The GLib-CRITICAL error applies to many applcations which call g_source_remove() with garbage parameters.

libglib2 changed recently to check for this error, instead of silently ignoring it, so the error started showing in multiple places.

The hang in consolekit happens independently of the call to g_source_remove() which is I suspect is simply a part of closing down the session.

Also: turns out: ssh was just coincidental
I am now able to reproduce the hang _without the GLib error_ in a local konsole session too.

To try and reprodce (not always, only happens intermittently)
1) Upgrade to 14.04
2) Open a konsole window and create several tabs in it
3) Run a command which is both CPU intensive on multiple cores, and generates output. I was much more successful reproducing when the command takes more than a minute to run and generates several hundred lines of output.
4) When the command finishes the konsole window tab in which the command was run no longer accepts input

[johnnyb0y (ubuntu64)]

Errors in my syslog only started after upgrade to 14.04. I noticed it immediately as a) I check my logs regularly and b) it's bulking out my syslogs in terms of size on disk

May 18 07:55:33 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1660 was not found when attempting to remove it
May 18 07:55:49 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1669 was not found when attempting to remove it
May 18 07:56:04 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1678 was not found when attempting to remove it
May 18 07:56:19 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1687 was not found when attempting to remove it
May 18 07:56:34 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1696 was not found when attempting to remove it
May 18 07:56:49 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1705 was not found when attempting to remove it
May 18 07:57:04 R2D2 console-kit-daemon[875]: GLib-CRITICAL: Source ID 1714 was not found when attempting to remove it

Occurs approximately once every 15 seconds on average and my system feels somewhat lethargic. Hope this gets fixed soon.
regards,
John G

[Ariel Faigon (ariel.faigon)]

Hi johnnyb0y,

Your system feeling lethargic is unlikely to be related to the GLib-CRITICAL warnings appearing in the logs.
They are just coincidental.

Check the output of `top` for heavy processes. In particular upgrade to 14.04 starts a very heavy new search indexing process called `/usr/bin/baloo_file` which I had to disable to get speed back. Check other bugs related to baloo for solutions.

hth

[Manfred Hampl (m-hampl)]

I assume that the GLIB-Critical messages can be avoided by adding some statements to reset the IDs in the source to prevent calling g_source_remove with an id that was already removed once:

e.g. for consolekit:
ck-job.c
        if (job->priv->out_watch_id > 0) {
                g_source_remove (job->priv->out_watch_id);
+ job->priv->out_watch_id = 0;
        }
        if (job->priv->err_watch_id > 0) {
                g_source_remove (job->priv->err_watch_id);
+ job->priv->err_watch_id = 0;
        }

ck-run-programs.c
        g_source_remove (cd->timeout_id);
+ cd->timeout_id = 0;

ck-tty-idle-monitor.c
        if (tty_idle_monitor->priv->timeout_id > 0) {
                g_source_remove (tty_idle_monitor->priv->timeout_id);
+ tty_idle_monitor->priv->timeout_id = 0;
        }

ck-vt-monitor.c
        if (vt_monitor->priv->process_queue_id > 0) {
                g_source_remove (vt_monitor->priv->process_queue_id);
+ vt_monitor->priv->process_queue_id = 0;
        }

(Not sure whether I correctly identified all cases where a variable should be reset after use.)
As already stated by others - the ssh hang most probably is not related to the GLIB critical error message noise.

[johnnyb0y (ubuntu64)]

I 'bit the bullet' and decided to remove consolekit (console kit daemon) with the following warnings:

sudo aptitude remove consolekit
The following packages will be REMOVED:
  consolekit
0 packages upgraded, 0 newly installed, 1 to remove and 1 not upgraded.
Need to get 0 B of archives. After unpacking 473 kB will be freed.
The following packages have unmet dependencies:
 cinnamon-session : Depends: consolekit but it is not going to be installed.
 lxsession-logout : Depends: consolekit but it is not going to be installed.
 libpolkit-qt-1-1 : Depends: consolekit but it is not going to be installed.
The following actions will resolve these dependencies:

     Remove the following packages:
1) cinnamon-session
2) libpolkit-qt-1-1
3) libqapt2-runtime
4) lxde
5) lxde-common
6) lxde-core
7) lxinput
8) lxsession
9) lxsession-logout

Accept this solution? [Y/n/q/?] y
The following packages will be REMOVED:
  alien{u} cinnamon-session{a} consolekit debugedit{u} galculator{u} giblib1{u} leafpad{u} libpolkit-qt-1-1{a} libqapt2-runtime{a} librpmbuild3{u} librpmsign1{u}
  libxmmsclient-glib1{u} lxappearance{u} lxde{a} lxde-common{a} lxde-core{a} lxinput{a} lxmusic{u} lxrandr{u} lxsession{a} lxsession-edit{u} lxsession-logout{a}
  lxshortcut{u} menu-xdg{u} rpm{u} scrot{u} xarchiver{u} xmms2-core{u} xmms2-plugin-alsa{u} xmms2-plugin-id3v2{u} xmms2-plugin-mad{u} xmms2-plugin-vorbis{u}
0 packages upgraded, 0 newly installed, 32 to remove and 1 not upgraded.
Need to get 0 B of archives. After unpacking 12.6 MB will be freed.
Do you want to continue? [Y/n/?] y

My system now responds much much quicker, my syslog doesn't get filled with errmsgs every 15 seconds and I'm happy, although my method of removal was fairly destructive so I wouldn't recommend it ;-)

My ssh connections remotely now work at the expected speed and I can now log in to my box remotely from work in a couple of seconds, instead of 10 seconds plus.

[Ariel Faigon (ariel.faigon)]

I figured the reason for my hangs was unrelated to the GLIB warning.
I had a sub process of the main job blocking on stdin input when arguments were missing, the main job was waiting for the sub process to finish so it didn't exit(1). Bottom line: the "hang" cause was a silly bug in one of my own scripts. Sorry about that.

So there's no hang issue in this bug - there's only a GLIB misuse and console warning.

I updated the bug description to be shorter/simpler accordingly.

I consider the severity/priority much lower now.

[Andrew Miles (admiles)]

I have this bug as well and its not an option for me to purge the consoleikit. My problems are not the syslog messages, its that ssh hangs for about 10 seconds before it lets me login. I have students running mpi jobs, launching them takes minutes rather than seconds if ssh could connect as it should.

I have verified that consolkit is in fact the problem by purging the consolekit, then ssh connections take a normal amount of time. Again this solution does not work for me as I need X11 logins with my x2go client.

If someone had a work-around or a fix that they could post that would be much appreciated.

[Manfred Hampl (m-hampl)]

I think I have given some hints on what source changes might be required in https://bugs.launchpad.net/ubuntu/+source/consolekit/+bug/1313042/comments/7

I am not sure if these changes will ever be implemented in the package, because http://freedesktop.org/wiki/Software/ConsoleKit/ tells
"ConsoleKit is currently not actively maintained. The focus has shifted to the built-in seat/user/session management of Software/systemd called systemd-logind!"

(I do not have any knowledge what has to be done to replace consolekit functionality by systemd-logind.)

[Andrew Miles (admiles)]

Thanks for that. I assume I would have to get the sources make the fixes and then recompile and then the error messages will go away.

Incidentally, in my case I am getting the consolekit log errors but my slow ssh logins were not caused by consolkit. I had a mis-configured DNS.