memory leak in unity-panel-service

[Andreas Büsching]

# Impact
The bad behavior of owncloud-client, combined with a design issue in Qt 5, and with unity-panel-service not cleaning up after old indicators, cause huge memory consumption of unity-panel-service. The consumption grows in arithmetic progression and can quickly reach several GBs.

For technical explanation of why this happens, see my comments #17 and #19.

I could not figure out how to explore the Unity panel contents in the GTK+ inspector. If that is possible, then the big row of the dead hidden indicators should be shown there.

# Proposed Fix
Fixing either unity-panel-service or owncloud-client will both work. In owncloud-client the fix is just one line, so it is easier. It disables the upstream workaround which was only needed for Qt 5.5. Upstream has also limited the workaround to Qt 5.5.0 only in newer code, so this patch will no longer needed with the next major owncloud-client release.

The same patch has been applied in Zesty version 2.2.4+dfsg-2ubuntu1.

# Regression Potential
Both bugs that the workaround took care of are fixed in Qt 5.6:
- https://bugreports.qt.io/browse/QTBUG-47863 — fixed in 5.5.1, 5.6.1
- https://bugreports.qt.io/browse/QTBUG-48068 — fixed in 5.6.0

So this should not cause any regressions.

# Test Case
Just start owncloud-client under Unity (and make sure it shows the tray icon). Logging in is not necessary, though that increases the memory consumption rate (as there are more menu items).

=============================================================================
I have seen similiar bug reports,but these are really old and are not related to version 16.10. This is way I'm opening a new one.

Currently the unity-panel-service process uses about 11GB of RAM which is definitely too much

top - 11:33:45 up 7 days, 4:09, 2 users, load average: 0,74, 1,08, 1,48
Tasks: 332 gesamt, 1 laufend, 331 schlafend, 0 gestoppt, 0 Zombie
%CPU(s): 3,7 be, 1,6 sy, 0,1 ni, 92,3 un, 2,2 wa, 0,0 hi, 0,0 si, 0,0 st
KiB Spch : 16309660 gesamt, 211552 frei, 14872856 belegt, 1225252 Puff/Cache
KiB Swap: 8266748 gesamt, 6138576 frei, 2128172 belegt. 609444 verfü Spch

  PID USER PR NI VIRT RES SHR S %CPU %MEM ZEIT+ BEFEHL
 4339 abuesch+ 20 0 11,590g 0,010t 12892 S 0,0 69,0 340:54.64 unity-panel-ser

Additionally to all standard indicators I have used the owncloud and cpufreq indicator. Don't know if this is related to it.

ProblemType: Bug
DistroRelease: Ubuntu 16.10
Package: unity 7.5.0+16.10.20160906.1-0ubuntu1
ProcVersionSignature: Ubuntu 4.8.0-22.24-generic 4.8.0
Uname: Linux 4.8.0-22-generic x86_64
.tmp.unity_support_test.0:

ApportVersion: 2.20.3-0ubuntu8
Architecture: amd64
BootLog: /dev/sda6: clean, 1281257/10911744 files, 22096310/43622400 blocks
CompizPlugins: No value set for `/apps/compiz-1/general/screen0/options/active_plugins'
CompositorRunning: compiz
CompositorUnredirectDriverBlacklist: '(nouveau|Intel).*Mesa 8.0'
CompositorUnredirectFSW: true
CurrentDesktop: Unity
Date: Fri Oct 21 11:27:57 2016
DistUpgraded: Fresh install
DistroCodename: yakkety
DistroVariant: ubuntu
DkmsStatus:
 virtualbox, 5.1.6, 4.8.0-22-generic, x86_64: installed
 virtualbox, 5.1.6, 4.8.0-26-generic, x86_64: installed
EcryptfsInUse: Yes
ExecutablePath: /usr/bin/compiz
GraphicsCard:
 Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller [8086:0126] (rev 09) (prog-if 00 [VGA controller])
   Subsystem: Dell 2nd Generation Core Processor Family Integrated Graphics Controller [1028:04a9]
InstallationDate: Installed on 2013-11-10 (1075 days ago)
InstallationMedia: elementary OS 0.2 "Luna" - Stable amd64 (20130810)
MachineType: Dell Inc. Latitude E6220
ProcEnviron:
 LANG=de_DE.UTF-8
 LANGUAGE=de_DE
 PATH=(custom, user)
 SHELL=/usr/bin/zsh
 XDG_RUNTIME_DIR=<set>
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-4.8.0-22-generic root=UUID=f1d78d25-23cf-4217-9db2-0750a8eb2b53 ro quiet splash vt.handoff=7
SourcePackage: unity
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 05/17/2012
dmi.bios.vendor: Dell Inc.
dmi.bios.version: A08
dmi.board.name: 0R97MN
dmi.board.vendor: Dell Inc.
dmi.board.version: A00
dmi.chassis.type: 9
dmi.chassis.vendor: Dell Inc.
dmi.modalias: dmi:bvnDellInc.:bvrA08:bd05/17/2012:svnDellInc.:pnLatitudeE6220:pvr01:rvnDellInc.:rn0R97MN:rvrA00:cvnDellInc.:ct9:cvr:
dmi.product.name: Latitude E6220
dmi.product.version: 01
dmi.sys.vendor: Dell Inc.
version.compiz: compiz 1:0.9.13.0+16.10.20160818.2-0ubuntu2
version.ia32-libs: ia32-libs N/A
version.libdrm2: libdrm2 2.4.70-1
version.libgl1-mesa-dri: libgl1-mesa-dri 12.0.3-1ubuntu2
version.libgl1-mesa-dri-experimental: libgl1-mesa-dri-experimental N/A
version.libgl1-mesa-glx: libgl1-mesa-glx 12.0.3-1ubuntu2
version.xserver-xorg-core: xserver-xorg-core 2:1.18.4-1ubuntu6
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev 1:2.10.2-1ubuntu1
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:7.7.1-1
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20160706-1ubuntu1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.12-2

[Andreas Büsching (crunchy)]

After about 2 days of usage the memory consumption is back to 9GB (started with around 60MB)

Additional the indicator menu do not work, meaning they are trying to draw their menu windows, but after a time they disappear

The CPU usage is up to about one full CPU (of 4). According to strace there is not that much happening (including sub-processes).

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Sebastian Heiden (seb-heiden)]

I'm having this issue too, the panel often freezes and gets unreponsive, even if the memory usage isn't that high. But after a while it fills up the RAM.

[Andreas Büsching (crunchy)]

Hi Sebastian,

Can you please click the 'affects me too' button to increase the chance for this bug to be investigated? thx in advance

[Andreas Büsching (crunchy)]

It looks like the memory leak in the process unity-panel-service has something to do with the ownclowd process. When I start the owncloud program it shows its indicator. About 2 hours later the process has grown from 80 MB to 130 MB.

[Robert Hutton (rwh-helms-deep)]

I am also running the owncloud sync client, version 2.2.4. unity-panel-service grows to over 4GB RAM in 4 hours.

[Gianluca Della Vedova (gianluca-dellavedova)]

I am also running the owncloud sync client.

[Sebastian Heiden (seb-heiden)]

Same, also running the owncloud sync client. Could be an bug in the owncloud package too.

[Jonas Schwabe (jonas-schwabe)]

Even if it was OwnCloud related, the memory usage should decrease after quitting OwnCloud...
Well it does not ;)

I had nearly 4gb yesterday and I'm back up toi 1gb right now.

[Gilles Gagniard (gilles-gagniard)]

I think I've got a working workaround :

- apt install appmenu-qt5
- in your ~/.profile, add 'export QT_QPA_PLATFORMTHEME=appmenu-qt5'
- reload your session

No more memory leak for me, but I've only applied that a couple of hours ago, so I'm not 100% sure yet.

If this is confirmed, it means the culprit is the new native Qt5 dbus appmenu/indicator support in yakkety, while in xenial and before an external plugin called 'appmenu-qt5' was used for the integration of Qt5 menus in Unity.

More information about this change here :
https://bugs.launchpad.net/ubuntu/+source/qtbase-opensource-src/+bug/1612767

Unfortunately it seems this old plugin is going the way of the dodo in zesty : https://bugs.launchpad.net/ubuntu/+source/appmenu-qt5/+bug/1634941

[Андрей Калинин (prize2step)]

Installing appmenu-qt5 resolves the problem with memory leak for me. Confirm: solution above works.

[Krzysztof Burliński (krzysztof-burlinski)]

Happens to me too, on 3 installations. I suspect owncloud or slack, but however this apps misbehave, it shouldn't get 2GB of memory on unity panel service.

Thanks for the workaround, i will give it a try

[Dmitry Shachnev (mitya57)]

Assigning to myself, though I do not know yet whether this is a bug in Unity or Qt. I will try to debug it later this week.

[Krzysztof Burliński (krzysztof-burlinski)]

Ok, the workaround helped me, at least for the RAM consumption issue.

Also, I noticed something like that, happening also since update to 16.10 (but I might be mistaken here)

All or some icons change to red signs:

http://i.imgur.com/BCVg6m8.png

not sure if it is connected with ram issue, but I might be helpful

[vojtabiberle (vojtech-biberle)]

I have same problem.

Two apps was responsible for higher RAM and MEM consumption:
 - ownCloud (https://github.com/owncloud/client)
 - Messenger for Desktop (https://github.com/Aluxian/Facebook-Messenger-Desktop)

Guys from ownCloud client found workaround in deny so many updates of icon, but I was not fully helpful (https://github.com/owncloud/client/issues/4985)

Workaround with appmenu-qt5 was helpful with RAM and MEM consumption but some apps have "no icon" icon and I thing, all of them are QT5 based.
 - RescueTime
 - ownCloud
 - Mega.co.nz
 - Albert

For me is main suspicious QT5 build-in plugin for icons.

[Dmitry Shachnev (mitya57)]

According to massif, the memory allocations come from the following path:

0x1FF8C3AA: new_item_normal (client.c:1095)
0x201A8798: menuitem_get_properties_new_cb (client.c:1578)
0x201A6C67: get_properties_callback (client.c:698)
0x06CB54E1: g_task_return_now (gtask.c:1121)
[...]

and the same, but new_item_separator instead of new_item_normal.

This is the libdbusmenu-glib / libdbusmenu-gtk code, so reassigning accordingly.

There is also going a very intensive D-Bus exchange between the indicator and the owncloud process, it did not look sane to me, so it may be another bug somewhere.

[James (jhesketh)]

I have this happening to me, I have seen up to 12GB being consumed by the Unity-Panel-Service. I did an upgrade from 16.04 to 16.10

I have Owncloud and Franz running as well.

[Dmitry Shachnev (mitya57)]

OK, after all I think it is an issue in unity-panel-service.

ownCloud does strange things: it unregisters and re-registers its tray icon periodically. This results in creating new indicator items every time. The new items are added to the menu by gtk_menu_shell_append call in panel_service_show_entry_common function. However, then the old indicator items get removed, the menu items are not removed (there is no corresponding gtk_container_remove call), they are just hidden by unsetting their entry2geometry hashes.

I hope Unity developers will be able to look at this more closely, and I will backport the upstream fix for ownCloud for the time being.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Dmitry Shachnev (mitya57)]

For Zesty fixed in owncloud-client 2.2.4+dfsg-2ubuntu1.

[Marco Trevisan (Treviño) (3v1n0)]

mitya57:

Thanks, we call gtk_widget_destroy on de-activation, but it was a little harder to see when that kind of behavior happened....

However, are you using a single-entry menu item with no menus? As we do it only in that case...
Do you have a simpler test case to reproduce this without the owncloud client?

I'll try it, though.

Thanks for the debugging hints.

[Marco Trevisan (Treviño) (3v1n0)]

Ah, also what's the upstream commit that fixed this?

[Brian Murray (brian-murray)]

Hello Andreas, or anyone else affected,

Accepted owncloud-client into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/owncloud-client/2.2.2+dfsg-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Dmitry Shachnev (mitya57)]

Hi Marco, thanks for looking at this!

Attached is a smaller test case. Compile it with:

g++ -fPIC -I /usr/include/x86_64-linux-gnu/qt5 -lQt5Core -lQt5Gui -lQt5Widgets ./test.cpp

To me, the chain of objects looks like this:

unity-panel-service top-level menubar (created in panel-service.c:2540)
-> menu item for every indicator entry, including dead ones
   (created in panel-service.c:2566, added in panel-service.c:2589)
-> item submenu (created in indicator-application.c:555, added in panel-service.c:2590)
-> dbusmenu client (created in libdbusmenu-gtk/menu.c:402)
-> finally, each dbusmenu client maintains its own menu structure
   (and all clients are alive and massively send D-Bus requests)

The upstream commit is https://github.com/owncloud/client/commit/98efb075357fa64f — it contained other changes, so I did not cherry-pick all of it. The first item in the commit message is relevant to us.

A question from me: is it possible to inspect unity-panel-service with GTK+ Inspector somehow? I tried running it with GTK_DEBUG=interactive, but the only object it shows to me is GtkSettings.

[Dmitry Shachnev (mitya57)]

I have tested indicator-applet on gnome-panel, it has the same issue. There the GTK Inspector works, and shows dead menu items as can be seen in the screenshot: http://people.ubuntu.com/~mitya57/inspector.png. I was also able to write a two-line fix for it (see the linked branch).

[Gilles Gagniard (gilles-gagniard)]

With owncloud-client from yakkety-proposed and appmenu-qt5 uninstalled :

$ dpkg -l | grep owncloud-client
ii owncloud-client 2.2.2+dfsg-1ubuntu0.1 amd64 folder synchronization with an ownCloud server - GUI
ii owncloud-client-data 2.2.2+dfsg-1ubuntu0.1 all ownCloudSync folder synchronization - shared data
ii owncloud-client-l10n 2.2.2+dfsg-1ubuntu0.1 all ownCloudSync folder synchronization - localization

I confirm the memory leak in unity-panel-service is gone.

*However*, it seems that there is now a bug with the indicator menu ... after a bit less than 24h running it, the menu is always empty (see attached screenshot). But owncloud-client is still synchronizing in the background.

[Dmitry Shachnev (mitya57)]

Hi Gilles, thanks for testing! Can you please do the following:

1) Wait until the menu becomes empty again;
2) Start “dbus-monitor 2>log.txt” in a terminal;
3) Wait a couple of minutes;
4) Kill dbus-monitor and attach log.txt to this bug (optionally gzip it if it’s too large).

That would help a lot, thanks in advance!

[Gilles Gagniard (gilles-gagniard)]

Here it is.

[Dmitry Shachnev (mitya57)]

Gilles, thanks for the log, it is very helpful!

Looks like the the tray menu is still rebuilt every 30 seconds, and Qt runs out of items IDs.

Backporting this upstream pull request should fix it: https://github.com/owncloud/client/pull/5072. I will do the new Zesty/Yakkety uploads tomorrow.

[Brian Murray (brian-murray)]

Hello Andreas, or anyone else affected,

Accepted owncloud-client into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/owncloud-client/2.2.2+dfsg-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Andreas Büsching (crunchy)]

The client is running for several hours now and has not increased its memory consumption a bit. The menu is still working.

Nice work!

[Morsok (morsok)]

Hello,

I'd like to test that also since I have to reboot my laptop every day due to the leak.

Can you link me to some documentation in order to switch from owncloud repos to the ubunto one ?

Also my owncloud client is 2.2.4 and the one you just uploaded is 2.2.2, is this normal ? Why the difference ?

What repository should be considered official ? The ubuntu one or the developers one ? Sorry if i'm asking dumb questions, this is still new to me.

[Dmitry Shachnev (mitya57)]

@Morsok: the owncloud repository is official from owncloud point of view. The Ubuntu repository is official from Ubuntu point of view — I am representing this one here :)

If you are already using 2.2.4 from owncloud repository, I will suggest you to remain on it and not downgrade, and wait for the fix from upstream instead (or use appmenu-qt5 as a workaround).

[Morsok (morsok)]

Hmmm ? But if both are officials, which one should a user choose ?

I guess upstream is the owncloud repository ? Do you have any information about the fix there ?

Thank you for taking the time I appreciate it !

[Dmitry Shachnev (mitya57)]

@Morsok: This is off-topic here. The issue is fixed in upstream Git, but I do not know when it is coming to their repository.

[Dmitry Shachnev (mitya57)]

Gilles, can you please check if your issue (menu becoming empty after some hours) is gone in the new version?

[Gilles Gagniard (gilles-gagniard)]

So far so good ! No reoccurrence for me since the latest update in yakkety-proposed.

[Dmitry Shachnev (mitya57)]

Marking as verified based on comments 33 and 39.

[Launchpad Janitor (janitor)]

This bug was fixed in the package owncloud-client - 2.2.2+dfsg-1ubuntu0.2

---------------
owncloud-client (2.2.2+dfsg-1ubuntu0.2) yakkety; urgency=medium

  * Backport upstream change to update the tray menu only on demand
    (update_menu_only_on_demand.patch). This should fix the remaining
    issues described in LP: #1635577.

 -- Dmitry Shachnev <email address hidden> Mon, 19 Dec 2016 18:37:49 +0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for owncloud-client has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.