notify-osd a memory hog (memory leak?)

[Rolf Leggewie]

Binary package hint: notify-osd

Noticed this today on my lucid box in top:
2647 myaccount 20 0 1196m 595m 3528 S 0 29.7 1:09.68 notify-osd

I don't think a simple notifcation daemon should suck up more than a Gig of memory of which half is real memory. It's the first time I noticed this. I don't have too much information at this point.

$ dpkg -l notify-osd*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-=============================================-=============================================-==========================================================================================================
ii notify-osd 0.9.29-0ubuntu2 daemon that displays passive pop-up notifications
ii notify-osd-icons 0.6 Notify-OSD icons

[Rolf Leggewie (r0lf)]

Glen, don't forget to actually ask a question if you have one.

[Glen Luscombe (glenluscombe)]

Hi Rolf,

Thanks for taking the time to report this bug and make Ubuntu better. Sorry for the late response, I'm doing this from my phone and it's not plain sailing...

Is it possible for you to run a Valgrind? Please follow the instructions at https://wiki.ubuntu.com/Valgrind, and attach the output to this bug report.

[Rolf Leggewie (r0lf)]

notify-osd runs as a daemon and thus is much harder to vaglrind. I'm not familiar with the process of how to do that. I'm not sure I have time to acquire the skills, but if you have more step-by-step instructions, I'm happy to have a shot at trying to reproduce it.

[Glen Luscombe (glenluscombe)]

My sincere apologies Rolf, I hadn't thought of the intricacies of running Valgrind on a daemon.

Does the memory leak occur soon after a reboot/login, or does it take time to accumulate?

I had a go myself at creating a Valgrind on notify-osd, and this is what worked for me:
1. killall notify-osd
2. G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=~/notify-osd-valgrind.log /usr/lib/notify-osd/notify-osd

You should be able to see notifications again once Valgrind has had time to start.

Then perhaps keep it running until you notice a large amount of memory in use, but this may be impractical for you if it takes days/weeks for the 1.2GB to accumulate (notify-osd is never more than 1.5MB in my system - but 106MB with Valgrind). Note as well, it will not show up in top/ps as notify-osd, but as memvheck-i386 or memcheck-amd64 depending on your distro (that might be helpful info too).

Then just kill it in the terminal with a Ctrl+C.

[Rolf Leggewie (r0lf)]

Glen, thank you for the detailed explanations. I ran them for a few days. While memory consumption did not go anywhere near my previous findings, it did increase steadily, starting out at 100M VIRT memory and ending up at 200M a few days later. It seems that saving images in firefox reproducably increased memory consumption. I attach the valgrind log in the hope it contains useful information.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Daniel Nyström (speakman)]

Lifting this bug since notify-osd is still leaking. Running for a few days and I'ts claiming 0.5GB RAM (on X86_64):
http://paste.ubuntu.com/5668799/

[Matthew Carpenter (matt-eisgr)]

Still leaking. I'm at 5.9gb and I just rebooted a few hours ago.
I had turned off KDE Notifications because plasma-shell was growing to huge amounts of RAM usage... so I wonder if there is some connection?

[Sybren Stüvel (sybren-stuvel)]

This also affects me, running Kubuntu 15.04:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 5197 sybren 20 0 16.398g 5.337g 13244 S 0.0 68.6 76:52.99 notify-osd

% uptime
 11:15:10 up 2 days, 21:44, 3 users, load average: 0.34, 0.56, 0.48

% apt-cache policy notify-osd
notify-osd:
  Installed: 0.9.35+15.04.20150126-0ubuntu1
  Candidate: 0.9.35+15.04.20150126-0ubuntu1
  Version table:
 *** 0.9.35+15.04.20150126-0ubuntu1 0
        500 http://nl.archive.ubuntu.com/ubuntu/ vivid/main amd64 Packages
        100 /var/lib/dpkg/status

[Q-collective (emiljacobs)]

I can confirm the bug in a fully updated instance of a 15.04 install (currently hogging 2GB out of 8GB of memory). As a workaround I just end the process, so it restarts with no memory usage. But this needs to be done every now and then, typically every day or so.

If logs are required, please ask. I'm not sure where notify-osd outut is logged, if at all.

[Hai NGUYEN VAN (psaxl)]

If local variable LOG is set to '1' then all logs shall appear in `~/.cache/notify-osd.log`. See: https://askubuntu.com/questions/288348/how-can-i-read-notifyosd-messages-after-they-are-displayed

[nh2 (nh2)]

This issue is still there in Ubuntu 16.04.

My /usr/lib/x86_64-linux-gnu/notify-osd grew to 812M resident usage (1289M VIRT) over the last days.

[nh2 (nh2)]

Well this already looks suspicious:

https://bazaar.launchpad.net/~indicator-applet-developers/notify-osd/trunk/view/505/src/stack.c#L813

    #define FORCED_SHUTDOWN_THRESHOLD 500
    ....

 // FIXME: this is a temporary work-around, I do not like at all, until
 // the heavy memory leakage of notify-osd is fully fixed...
 // after a threshold-value is reached, "arm" a forceful shutdown of
 // notify-osd (still allowing notifications in the queue, and coming in,
 // to be displayed), in order to get the leaked memory freed again, any
 // new notifications, coming in after the shutdown, will instruct the
 // session to restart notify-osd
 if (bubble_get_id (bubble) == FORCED_SHUTDOWN_THRESHOLD)
  g_timeout_add (defaults_get_on_screen_timeout (self->defaults),
          _arm_forced_quit,
          (gpointer) self);

[nh2 (nh2)]

I think I have the solution.

First an explanation of the above quitting logic: The "bubble IDs" count upwards from 1, one for each notification bubble shown. When the bubble ID reaches FORCED_SHUTDOWN_THRESHOLD = 500, then _arm_forced_quit() is called, which quits the process using gtk_main_quit() and thus terminates notify-osd, and it is restarted automatically (I think by DBUS) when the next notification comes in.

So the first bubble ID is 1, the second 2, and so on.
You can verify that e.g. by inserting

 printf("bubble ID %u\n", bubble_get_id (bubble));

But this is only the case if you wait between your `notify-send mytext` invocations until the current bubble has faded out and is no longer shown.
If you call `notify-send mytext` while the previous notification is still being displayed then bubble ID is actually 0.
(This is because the variable `bubble` is NULL in that case, and bubble_get_id(NULL) returns 0.)
But the bubble ID generation counter is still incremented nevertheless.
So if your generated bubble IDs are 1, 2, and then 0, the next bubble ID generated will be 4.

You can generate this situation e.g. this way (note that the default notification timeout is 10 seconds, so I sleep 11 seconds):

    notify-send test && sleep 11 && notify-send test && notify-send test && sleep 11 && notify-send test

So when you send notifications in quick succession, their bubble IDs are skipped.

Now, if you send them in quick succession around the 500th notification, then the check

  if (bubble_get_id (bubble) == FORCED_SHUTDOWN_THRESHOLD)

will never trigger (e.g. because the bubble IDs will be 499, 0, and 501).

Then the forced shutdown will never happen, and notify-osd will leak memory forever instead of being restarted every 500th notification.

The fix is to use

  if (bubble_get_id (bubble) >= FORCED_SHUTDOWN_THRESHOLD)

instead.

[nh2 (nh2)]

Here's a patch to fix the issue.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "Patch fixing the shutdown limit being exceeded" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[nh2 (nh2)]

Hey, is there any interest in getting this fixed?

Be reminded that due to this memory leak, notify-osd can easily take up a 1 GB of memory after the computer runs for a while. Just yesterday I had to manually kill it again in this situation.

Most laptops out there these days likely run on 4 GB RAM or less.

Spending 25% of an average user's machines on notify-osd's memory leak seems like something that should be addressed quickly, especially if the fix is trivial.

[Rolf Leggewie (r0lf)]

nh2, thank you for your work. It's much appreciated.

I'm backlogged and thus have only become aware of your contribution now. Sorry about that. Have you talked to upstream about your patch? Looks like an easy fix after all (I'm sure that finding it was hard work, thank you for that).

[Rolf Leggewie (r0lf)]

Unfortunately, notify-osd seems to be abandoned. Not sure which project is the replacement. We should still try to get the patch packaged.