nm-applet leaks memory

[Harm van Bakel]

The nm-applet embedded in the task bar becomes indeterminestically unresponsive. Clicking on the applet will still show the popup menu, but none of the menu entries respond to mouse clicks. When this happens, the "VPN connections" and "Other networks" menus also show up completely empty (only a small menu stub is displayed).

It is uncertain what causes this behavior. It was originally thought that the cause was memory leaks in nm-applet: nm-applet leaks memory in a fresh install of Natty, increasing to over 200 Mb in 6-8 hours. However, fixes for the memory leaks failed to resolve the occurrence of the anomalous behavior described above.

dbus-monitor shows errors such as the following when clicking a menu item or attempting to access a submenu:
error sender=:1.14 -> dest=:1.67 error_name=org.gtk.GDBus.UnmappedGError.Quark._LIBDBUSMENU_2dGLIB.Code0 reply_serial=107705
   string "The IDs supplied '[80846]' do not refer to any menu items we have"

The errors are returned by bus_event_group or bus_about_to_show_group in libdbusmenu-glib/server.c.

--

Stable Release Update Application:

[Impact]
This issue affects most and any users of nm-applet, especially in environments (such as offices) where the detected wireless networks change a lot, and where roaming can occur frequently.

[Test Case]
Run nm-applet for multiple hours:
 - Observe that the VPN Connections submenu still opens and lists connections (if VPN connections are configured)
 - Observe that the "More networks" submenu for additional detected wireless networks still opens and lists networks.

A common failure case is where such a submenu will open but show an empty list (a piece of menu a few milimeters long).

[Regression Potential]
Minimal, the fixes have been available for a while in the development release (and in other recent releases) with no regressions; furthermore, this corrects "obviously" wrong memory handling without changing the effective behavior of the application.
Possible issues could be introducing new memory leaks with the changes however, and although the risk is minimal, this could cause the same failure behavior as listed above under [Test Case].

--

ProblemType: Bug
DistroRelease: Ubuntu 11.04
Package: network-manager-gnome 0.8.4~git.20110318t152954.9c4c9a0-0ubuntu1
ProcVersionSignature: Ubuntu 2.6.38-9.43-generic-pae 2.6.38.4
Uname: Linux 2.6.38-9-generic-pae i686
NonfreeKernelModules: nvidia
Architecture: i386
CRDA: Error: [Errno 2] No such file or directory
Date: Tue May 10 11:53:09 2011
EcryptfsInUse: Yes
IfupdownConfig:
 auto lo
 iface lo inet loopback
InstallationMedia: Ubuntu 11.04 "Natty Narwhal" - Beta i386 (20110413)
IpRoute:
 192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.12 metric 1
 169.254.0.0/16 dev eth0 scope link metric 1000
 default via 192.168.2.1 dev eth0 proto static
Keyfiles: Error: [Errno 2] No such file or directory
NetworkManager.state:
 [main]
 NetworkingEnabled=true
 WirelessEnabled=true
 WWANEnabled=true
ProcEnviron:
 LANGUAGE=en_CA:en
 PATH=(custom, user)
 LANG=en_CA.UTF-8
 SHELL=/bin/bash
SourcePackage: network-manager-applet
UpgradeStatus: No upgrade log present (probably fresh install)

[Bartosz Fenski (fenio)]

I don't use any VPNs nor other networks, simply Wifi networks and this is what I got after 9 days of uptime:

fenio 1827 0.0 32.2 968800 633852 ? SLl May01 5:32 nm-applet --sm-disable

633MBs of RAM.

And on my girlfriend's laptop after 4 days 158MB.
sylwia 1602 0.0 15.6 321632 158792 ? Sl May06 3:14 nm-applet --sm-disable

It's quite obvious that there has to be some huge memory leak somewhere.

regards
fEnIo

[Harm van Bakel (hvbakel)]

Definitely seems linked to WiFi network polling since it occurs even when I'm not connected to any wireless network and the memory leak and problems dissappear when I disable the wireless network card alltogether. The rate of the memory leak seems to increase with the number of wireless networks in the neighborhood.

[JKL (jkl102001)]

I am working on this. I have found 4 separate bugs so far. There are more however. Note the first bug listed below was closed, but the problem was not fixed. It needs to be reopened by someone who has the authority. I can neither reopen old bugs nor can I actually commit anything. I hope someone on the Ubuntu end will help.

https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/724554

https://bugs.launchpad.net/libappindicator/+bug/784327

https://bugs.launchpad.net/ubuntu/+source/libgnome-keyring/+bug/784369

https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/784711

All I am doing is running nm-applet under valgrind and reading code. Activating and deactivating my wireless radio seems to reliably push memory usage up. The problem is of course sifting through all the garbage from gtk in the logs. If anyone has an up-to-date suppressions file it would be useful. The official one over on gnome live seems to be a dead link.

[Harm van Bakel (hvbakel)]

@JKL: Thanks for looking into this, much appreciated. If I might offer a suggestion, if you are able to patch some of these bugs and upload a modified version of the applet in a ppa, I would be more than happy help to test it out.

[JKL (jkl102001)]

Here is another bug.

https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/784756

I'll let everyone know if I publish any updated packages. I'm hoping an actual package maintainer will step in here at some point however.

A workaround for now is to kill nm-applet and restart it. You could do this first thing in the morning every day maybe. When you restart it, do

nm-applet --sm-disable

You can just do this using your own user account. Root privileges are not needed. Alternatively, I think logging out and logging back in should restart the applet.

[JKL (jkl102001)]

Bug number 6:

https://bugs.launchpad.net/ubuntu/+source/libgnome-keyring/+bug/784788

[Mathieu Trudel-Lapierre (cyphermox)]

An actual package maintainer steps in here regularly ;)

JKL, thanks for looking into this, I'll review the attached patches or fix the code (or you can continue to provide patches, I'll happily include them).

[JKL (jkl102001)]

Here is the seventh. This one is in dbusmenu.

https://bugs.launchpad.net/dbusmenu/+bug/784808

[JKL (jkl102001)]

Number eight. This is debusmenu again, and the problem is similar to the previous one.

https://bugs.launchpad.net/dbusmenu/+bug/784873

[JKL (jkl102001)]

Number nine.

https://bugs.launchpad.net/dbusmenu/+bug/784890

[JKL (jkl102001)]

So far the leaks identified in the bugs I filed above are relatively small, and they don't account for the massive leaks I am seeing. Each time I disable and then re-enable the wireless using hotkeys, it causes nm-applet to consume between 300K and 500K of additional memory. The memory grows at a regular rate even if I do nothing, and in just a few minutes it jumped another MB or so.

The leak identified in log entry below is on the right order of magnitude, but I cannot tell where the leak actually is.

==10301== 111,377 (5,520 direct, 105,857 indirect) bytes in 138 blocks are definitely lost in loss record 9,321 of 9,326
==10301== at 0x4C28FAC: malloc (vg_replace_malloc.c:236)
==10301== by 0x8F62A62: g_malloc (gmem.c:164)
==10301== by 0x8F79666: g_slice_alloc (gslice.c:842)
==10301== by 0x8F97A09: g_variant_alloc (gvariant-core.c:475)
==10301== by 0x8F97B0C: g_variant_new_from_children (gvariant-core.c:560)
==10301== by 0x8F9532D: g_variant_builder_end (gvariant.c:3260)
==10301== by 0x6D1D9A7: parse_value_from_blob (gdbusmessage.c:1447)
==10301== by 0x6D1DABE: parse_value_from_blob (gdbusmessage.c:1431)
==10301== by 0x6D1F0BD: g_dbus_message_new_from_blob (gdbusmessage.c:1781)
==10301== by 0x6D27FF4: _g_dbus_worker_do_read_cb (gdbusprivate.c:737)
==10301== by 0x6CD6D48: complete_in_idle_cb (gsimpleasyncresult.c:757)
==10301== by 0x8F5BBCC: g_main_context_dispatch (gmain.c:2440)
==10301== by 0x8F5C3A7: g_main_context_iterate.clone.6 (gmain.c:3091)
==10301== by 0x8F5C9F1: g_main_loop_run (gmain.c:3299)
==10301== by 0x6D26C43: gdbus_shared_thread_func (gdbusprivate.c:276)
==10301== by 0x8F833E3: g_thread_create_proxy (gthread.c:1897)
==10301== by 0x86A3D8B: start_thread (pthread_create.c:304)
==10301== by 0x92EE04C: clone (clone.S:112)

This thread is started when a g_dbus_worker is created.

http://git.gnome.org/browse/glib/tree/gio/gdbusprivate.c

For some reason dbus messages (message bodies in particular) are not getting cleaned up properly.

I checked Fedora 15, and this problem does not happen there. So the problem must be in the Ubuntu-specific modifications to nm-applet (appindicator, etc.).

[JKL (jkl102001)]

The core problem is definitely in appindicator or something appindicator depends on. I wrote a small test program that updates an appindicator menu once a second, similar to what nm-applet does.

It exhibits the same leak in the dbus message system that I posted above.

[JKL (jkl102001)]

Got it. The bug is all the way down in glib.

https://bugzilla.gnome.org/show_bug.cgi?id=629684

Patch submitted.

[JKL (jkl102001)]

Number eleven.

==10301== 144 bytes in 9 blocks are definitely lost in loss record 8,163 of 9,326
==10301== at 0x4C28FAC: malloc (vg_replace_malloc.c:236)
==10301== by 0x8F62A62: g_malloc (gmem.c:164)
==10301== by 0x8F79666: g_slice_alloc (gslice.c:842)
==10301== by 0x8F7A992: g_slist_append (gslist.c:254)
==10301== by 0x65862CA: dbus_g_connection_register_g_object (in /usr/lib/libdbus-glib-1.so.2.1.0)
==10301== by 0x527094B: nm_settings_service_export_connection (nm-settings-service.c:241)
==10301== by 0x4463EA: internal_add_connection (nma-gconf-settings.c:113)
==10301== by 0x4465D6: read_connections (nma-gconf-settings.c:244)
==10301== by 0x44665E: list_connections (nma-gconf-settings.c:270)
==10301== by 0x52701D1: impl_settings_list_connections (nm-settings-service.c:107)
==10301== by 0x526FEBB: dbus_glib_marshal_nm_settings_BOOLEAN__POINTER_POINTER (nm-settings-glue.h:97)
==10301== by 0x6584C4C: ??? (in /usr/lib/libdbus-glib-1.so.2.1.0)
==10301== by 0x8475A00: _dbus_object_tree_dispatch_and_unlock (dbus-object-tree.c:858)
==10301== by 0x8467B0F: dbus_connection_dispatch (dbus-connection.c:4688)
==10301== by 0x6582654: ??? (in /usr/lib/libdbus-glib-1.so.2.1.0)
==10301== by 0x8F5BBCC: g_main_context_dispatch (gmain.c:2440)
==10301== by 0x8F5C3A7: g_main_context_iterate.clone.6 (gmain.c:3091)
==10301== by 0x8F5C9F1: g_main_loop_run (gmain.c:3299)
==10301== by 0x416D77: main (main.c:101)

It looks like this one is covered in one of these two upstream bugs already filed.

https://bugs.freedesktop.org/show_bug.cgi?id=36793
https://bugs.freedesktop.org/show_bug.cgi?id=36811

[JKL (jkl102001)]

My analysis was mistaken in comments 12-14, which discussed a leak I was thinking of as number ten. The remote bug watch for glib should be removed. I tried, but I got this:

Oops!

Sorry, something just went wrong in Launchpad.

We’ve recorded what happened, and we’ll fix it as soon as possible. Apologies for the inconvenience.

(Error ID: OOPS-1968AR579)

[JKL (jkl102001)]

Number twelve.

https://bugs.launchpad.net/ubuntu/+source/appmenu-gtk/+bug/787736

[Mathieu Trudel-Lapierre (cyphermox)]

Thanks for all the work in analysing the valgrind logs and figuring out the issues.

I'm marking this Triaged, since it's pretty clear what needs to be done (thanks to all the bug reports), and High given the relatively high impact across at least the indicator apps.

After going through the bugs filed; I think we're left to three that need to be Triaged and the patches written or reviewed:
https://bugs.launchpad.net/ubuntu/+source/appmenu-gtk/+bug/787736
https://bugs.launchpad.net/ubuntu/+source/libdbusmenu/+bug/784890
https://bugs.launchpad.net/ubuntu/+source/libappindicator/+bug/784327

Aside from that, dbus-glib 0.94 may need to be investigated, or at least backporting the patches from the 0.94 branch onto 0.92 which we currently have in Oneiric.

[JKL (jkl102001)]

Bug 784890 (the reference leaks in g_variant_get_child in particular) is what is causing the large leak I described in comment 12. The tricky part is that there are two different threads involved, so it was hard to track down the cause.

I patched all of these issues locally for purposes of testing, so that valgrind no longer reports any "definitely lost" errors, but I still see increasing memory use in nm-applet. The memory growth is on the order of 1MB per day under normal use, and I still see memory use increase by about 200KB when I disable then re-enable the wireless radio.

I can think of three possible reasons for this:

1. There are memory leaks in reachable memory. Improperly cleaned-up circular references could cause this.

2. Internal data structures such as hash tables grow in size over time.

3. Glibc isn't returning free memory back to the OS, perhaps due to memory fragmentation.

[demilord (demilord)]

Damn , isnt it better ubuntu moves to a other program like indicator-network..
Its been there for more then a year and still not solved the leaks...

[Jeremy (jeremybmerrill)]

I have the same bug.

Not sure what other info would be helpful...

After 16 hours:
~$ ps aux | grep nm-applet
<username> 15089 0.1 7.7 279724 160220 ? SLl Jun30 1:59 nm-applet
<username> 19538 0.0 0.0 4156 848 pts/2 S+ 13:41 0:00 grep nm-applet

I'm running pidstat, and interestingly, memory use was climbing steadily (>1mb/10min) at home with a WPA-encrypted connection but climbing more slowly not that I'm on a WEP-encrypted connection.

I'm happy to help provide info if anyone needs it (I'm subscribed); this bug has been seriously interfering with my use of Ubuntu, since it will use up so much memory that it starts using swap and I either have to kill the power or wait 30 minutes for to get enough control to kill nm-applet...

[Martin Spacek (mspacek)]

I just marked bug #748661 as a duplicate of this one. I hope this is the right place to complain about the "More networks" and "VPN" menus being just empty stubs. I've posted a couple of screenshots in the aforementioned bug.

[Martin Spacek (mspacek)]

I suspect bug #779754 is a duplicate of this as well.

[JazZ (misterj-home)]

I confirm this bug. nm-applet takes somethins like 300Mo after 3 days without shuting down the computer. This leak additioned with e-calendar-factory leak is a bit too much !

[Mathieu Trudel-Lapierre (cyphermox)]

Marking as a duplicate of bug 779754.

[Martin Spacek (mspacek)]

I've been living with this bug for nearly a year. Luckily, I have 16GB of RAM, but I still need to restart nm-applet every few days when the more networks and VPN submenus stop working (which correlates very well to extreme memory usage).

Are there no patches that can be combined/backported to finally fix this in Natty? Are there any new insights from Precise in Bug #930491 that might apply to Natty? Does anyone have a PPA I can try?

[Martin Spacek (mspacek)]

Also, maybe

[Martin Spacek (mspacek)]

Oops. Also, maybe this should be marked as a duplicate of Bug #684599 ? That one's a fair bit older, and has a lot more activity on it.

[Mathieu Trudel-Lapierre (cyphermox)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. The issue that you reported is one that should be reproducible with the live environment of the Desktop CD of the development release - Precise Pangolin. It would help us greatly if you could test with it so we can work on getting it fixed in the next release of Ubuntu. You can find out more about the development release at http://www.ubuntu.com/testing/ . Thanks again and we appreciate your help.

[Martin Spacek (mspacek)]

I tried live usb booting into Precise amd64 a few days before final release, left it running overnight while bittorrenting. Didn't notice much of an increase in memory usage in nm-applet, maybe 10% at most, which is far far better than what I'm seeing in natty. Also, I never encountered the little menu stub that's common in Natty after a long enough time. So things have obviously improved. The version in Precise (0.9.4.1 ish) is a fair bit newer than the one in Natty (0.8.4 ish). Again, any chance some patch(es) might get backported? This really is a security issue, in a networking component no loess, and Natty is supposed to remain supported until October 2012.

[Paul Dwerryhouse (paul-dwerryhouse)]

This bug also affects Precise; I encounter it regularly, with the same symptoms as described above - the icon remains, but becomes unresponsive. The only way to fix it is to kill nm-applet and restart it.

[Christopher Townsend (townsend)]

We are also seeing this again in Precise. It was gone in Oneiric, so it seems there has been a memory leak regression introduced.

[Christopher Townsend (townsend)]

I should also note that this is being seen with network-manager-applet package version 0.9.4.1-0ubuntu2 in Precise.

[Howard Chu (hyc)]

Just echoing comment #33. My nm-applet was over *1.3GB* after only 7 days of uptime.

Very Unhappy Camper here.

[James M. Leddy (jm-leddy)]

I'm not seeing this bug. After 2 days of uptime, nm-applet is only using 16M for RSS. Would someone who is experiencing the issue please give more context on how to reproduce?

[Tom Fredrik Blenning Klaussen (bfg-dev)]

I haven't been able to verify this, but in my hunch is on unstable networking. Eg. connection is torn down and reestablished often.

I'm in an apartment building with countless networks in the proximity, which causes the connection to be reestablished quite frequently.

[James M. Leddy (jm-leddy)]

I'm seeing the original bug, where you can't select from the new list of networks. However, it doesn't seem to be a memory leak, since nm is only at 1% of my systems memory right now.

[Marlon Nelson (marlon-nelson)]

i still have this problem, which i use the following alias to work around it

# alias renm='killall nm-applet ; nm-applet &'

a PITA but it gets me through the day

the problem exists on 2 of my computers which use WiFi. the other computer
is hooked directly to my router via an ethernet cable and i don't recall
seeing the problem on it

i also live in an apartment with many networks... 25 at the moment

is there a way to detect & count how many times the connection is torn down
and reestablished during the day?

On Mon, Aug 20, 2012 at 2:51 PM, Tom Fredrik Blenning Klaussen <
<email address hidden>> wrote:

> I haven't been able to verify this, but in my hunch is on unstable
> networking. Eg. connection is torn down and reestablished often.
>
> I'm in an apartment building with countless networks in the proximity,
> which causes the connection to be reestablished quite frequently.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/780602
>
> Title:
> nm-applet leaks memory and stops functioning after a while
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/oem-priority/+bug/780602/+subscriptions
>

--
-eom-

[Tom Fredrik Blenning Klaussen (bfg-dev)]

I suppose I can have a go at running nm-applet in valgrind. I'll try to get around to it in a few days.

[Tom Fredrik Blenning Klaussen (bfg-dev)]

Second thought, the binaries are probably stripped and without debug-symbols. Is there an easy way to get the correct binaries with symbols?

[James M. Leddy (jm-leddy)]

Personally, I'm not convinced this is 'regression-released'. I seem to remember a time when this used to work flawlessly on 12.04. I'm going to downgrade and see if that helps at all.

[James M. Leddy (jm-leddy)]

Okay, so there is no downgrade for precise since nm was never updated. I seem to be able to reproduce the issue by leaving nm running a long time, like it just happened since nm has been running overnight. This leads credance to the memory leak theory, but still I only have 12m RSS which doesn't seem like that much.

I've recompiled the quantal packages for precise and will see if this fixes the issue. If so, we'll probably have to bisect the change that fixed this problem.

[James M. Leddy (jm-leddy)]

This is still an issue with the version of nm-appplet in quantal. I'm attaching the massif file for analysis. This with version 0.9.6.2-0ubuntu2

[James M. Leddy (jm-leddy)]

There was an issue with the massif file in that I didn't have debug packages installed. I'm retrying with the proper debug packages installed now.

[James M. Leddy (jm-leddy)]

More massif data

[Martin Spacek (mspacek)]

I've switched from natty (Ubuntu 11.04) to a fresh install of quantal (Xubuntu 12.10), so I'm now running network-manager 0.9.6.0-0ubuntu7 and network-manager-gnome 0.9.6.2-0ubuntu6 (not sure why they're different versions). It looks like the massive memory leaks in nm-applet are gone. nm-applet tends to hover around 6MB of memory usage. However, after running for a couple of days, the applet stops responding. I think, though I'm not sure, that network connectivity eventually breaks some time later.

As before, when nm-applet becomes unresponsive, I can still click the nm-applet icon in the indicator plugin in the panel to bring up the menu, but the VPN submenu only shows a stub, clicking on "Information" or "Edit" does nothing, and I suspect clicking on any of the other available networks doesn't do anything either. Killing and then restarting nm-applet fixes it for another couple of days. As many others, I'm always on WiFi, I suspend and resume often, and there tend to be a lot of wireless networks within range. Which, if any, of these factors makes a difference in causing nm-applet to become unresponsive remains unknown.

Has anyone tried the latest 0.9.6.4 release from http://projects.gnome.org/NetworkManager ? It has some hopeful points about stability in the release notes:

http://ftp.gnome.org/pub/GNOME/sources/NetworkManager/0.9/NetworkManager-0.9.6.4.news

[Alex Chiang (achiang)]

This is still extremely reproducible on 12.04.

achiang@yew:~$ uptime
 11:32:29 up 2 days, 2:22, 7 users, load average: 0.23, 0.26, 0.33

achiang@yew:~$ smem -k -t
  PID User Command Swap USS PSS RSS
 1930 achiang /usr/lib/indicator-session/ 1.0M 448.0K 481.0K 3.2M
 1933 achiang /usr/lib/indicator-applicat 688.0K 632.0K 654.0K 3.1M
 1924 achiang /usr/lib/indicator-datetime 1.1M 732.0K 768.0K 3.6M
 1925 achiang /usr/lib/indicator-sound/in 1.1M 792.0K 859.0K 3.7M
 1928 achiang /usr/lib/indicator-printers 1.4M 1.5M 1.7M 7.2M
 1929 achiang /usr/lib/indicator-messages 580.0K 1.9M 2.0M 5.1M
 4689 achiang /usr/lib/notify-osd/notify- 16.0K 4.1M 4.4M 11.2M
 1861 achiang nm-applet 2.6M 7.6M 8.0M 15.6M
 1981 achiang /usr/lib/indicator-appmenu/ 2.0M 10.9M 11.0M 13.5M
 1851 achiang unity-2d-panel 6.3M 16.8M 18.3M 26.8M

Look at nm-applet vs the other indicators. 15.6M of RSS vs the others which is like ~3M seems really wrong.

[James M. Leddy (jm-leddy)]

I've uploaded the valgrind log from this:

 G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log nm-applet

I've got a bit of a Frankenstein system in that I'm using the Quantal nm-applet on Precise. So if it's a problem in one of the deps it's possible. Though if comment #46 is any indication it's still a problem in Quantal as well. I'm planning on upgrading real soon now, so I'll test once I have time.

[James M. Leddy (jm-leddy)]

Currently for valgrind output, I've installed the following packages:

ii libc6-dbg 2.15-0ubuntu10.3 Embedded GNU C Library: detached debugging symbols
ii libglib2.0-0-dbg 2.32.3-0ubuntu1 Debugging symbols for the GLib libraries
ii libgtk-3-0-dbg 3.4.2-0ubuntu0.4 GTK+ libraries and debugging symbols
ii libgtk2.0-0-dbg 2.24.10-0ubuntu6 GTK+ libraries and debugging symbols
ii network-manager-dbg 0.9.6.0-0ubuntu5 network management framework (debugging symbols)
ii network-manager-pptp-dbgsym 0.9.6.0-0ubuntu1 debug symbols for package network-manager-pptp

[Alex Chiang (achiang)]

Thanks James. In addition, for 12.04, I've also installed

network-manager-gnome-dbg
libpango1.0-0-dbg

[Alex Chiang (achiang)]

It took a little while but I was able to reproduce this under valgrind in 12.04.

achiang@yew:~$ G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log --track-origins=yes nm-applet
** Message: applet now removed from the notification area

** (nm-applet:13124): CRITICAL **: nm_secret_agent_register: assertion `priv->reg_call == NULL' failed

** (nm-applet:13124): WARNING **: Failed to start applet secret agent!

(nm-applet:13124): GdkPixbuf-CRITICAL **: gdk_pixbuf_composite: assertion `dest_y >= 0 && dest_y + dest_height <= dest->height' failed
^C** Message: PID 0 (we are 13124) sent signal 2, shutting down...

[Alex Chiang (achiang)]

After inspecting the valgrind log and reading a bunch of source, I found the problem in 12.04.

http://bazaar.launchpad.net/~network-manager/network-manager-applet/ubuntu/revision/364

Needs to be backported/SRU'ed into 12.04.

[Alex Chiang (achiang)]

I should say that at least I found one problem, but it might not fix all the problems. :)

[Jeffrey Knockel (jeff250)]

Alex, thanks for taking a stab at this. I've built and installed the 0.9.4.1-0ubuntu3 packages from your bzr branch, and then I rebooted. Unfortunately, after running for ~24 hours, I can still reproduce the unresponsive menu entries and empty "More networks" and "VPN Connections" menus that are described in the original bug description.

[Alex Chiang (achiang)]

Jeffrey, did you install all the packages or just some of them? I installed:

ii libnm-gtk-common 0.9.4.1-0ubuntu3~achiang1 network management framework (common files for wifi and mobile)
ii libnm-gtk0 0.9.4.1-0ubuntu3~achiang1 network management framework (GNOME dialogs for wifi and mobile)
ii network-manager-gnome 0.9.4.1-0ubuntu3~achiang1 network management framework (GNOME frontend)
ii network-manager-gnome-dbg 0.9.4.1-0ubuntu3~achiang1 network management framework (debugging symbols)

If you did install all the packages, any chance you could re-run under valgrind? It's not too hard to do...

$ apt-get install valgrind
$ killall nm-applet
$ G_SLICE=always-malloc G_DEBUG=gc-friendly valgrind -v --tool=memcheck --leak-check=full --num-callers=40 --log-file=valgrind.log --track-origins=yes nm-applet

And then let that run until you can reproduce. Then attach the valgrind.log here.

[Alex Chiang (achiang)]

This is my valgrind log after using my patched packages for ~12 or so hours. I wasn't able to observe the broken behavior, and additionally, the leaks are no longer reported in the log.

We still do have a fontconfig leak, but that's unrelated here.

[Mathieu Trudel-Lapierre (cyphermox)]

I targetted the proper releases... FWIW, the leaks are most likely fixed in raring and quantal -- Alex, can you confirm this?

Marking the Precise task as In Progress, I'm about to merge and upload the package from Alex.

Alex, can you please also report the fontconfig leak in a separate bug? It really ought to be fixed, it's long overdue.

Next step is just to update description for the SRU paperwork.

[Jeffrey Knockel (jeff250)]

Alex, yes, I've installed all *.deb from build-area.

I'm attaching a python spam script that I modified from the examples network-manager comes with that can reproduce the issue for me in ~1 minute by repeatedly adding and deleting a connection via dbus. I see the bug appear after about 500 iterations, but I have > 50 wifi networks visible in my menu at the moment, so it might take others many more iterations if we think that's relevant. During running the script, if you open the nm-applet menu, you should see it flickering madly. (Sometimes the script will stall for a few seconds due to a dbus timeout--haven't debugged this yet, gah...)

I can't reproduce it using this spam script running under valgrind though, probably due to valgrind's performance penalty (the menu doesn't visibly flicker when I run the script). I'll try to get a valgrind log the more natural way by just letting it run, which hopefully will still take ~24-48 hours, assuming I don't have to reboot for anything in the meantime.

[Alex Chiang (achiang)]

Mathieu, yes, I confirm the leaks are fixed in quantal and raring via code inspection (but not actual testing). Additionally, I did some digging into fontconfig and it looks like they are false positives since 2008. I'm experimenting with a valgrind suppression file so we don't have to deal with these in the future.

http://lists.freedesktop.org/archives/fontconfig/2008-November/003032.html

Jeffrey, nice script. I notice in my valgrind log that although the nm-applet leaks are gone, there are still a few dbus and other leaks, which makes sense given the context of your script. I bet they haven't all been backported to 12.04 yet. Something to keep investigating...

[Jeffrey Knockel (jeff250)]

Alex, I'm attaching my valgrind log from running nm-applet for about ~18 hours, terminating with SIGINT when I saw the empty menu lists failing the [Test Case]. This log was generated *without* running my spam script. This log was generated running the *ubuntu2.1 packages from the bzr branch merged yesterday, in case they are substantially different from the *ubuntu3 packages you had before.

[Alex Chiang (achiang)]

Thanks Jeff. Clearly there are more leaks that haven't been fixed yet.

Here's a debdiff for 12.04's appmenu-gtk. I don't think this will fix the nm-applet bug by itself, but it will help.

If you could try testing it too, that would be great, thanks.

[Jeffrey Knockel (jeff250)]

Thanks Alex. Unfortunately, after ~24 hours, I still see the nonresponsive menu entries and the empty menus with your patched version of the appmenu-gtk* packages.

[James M. Leddy (jm-leddy)]

I just ran into this bug today again with nm-applet in the latest precise devel tree. I've compiled again against appmenu-gtk with achaing's patch in bug 787736. I'm running massif on the resulting binary and will attach to the bug if I nm-applet stops being functional again.

[Mathieu Trudel-Lapierre (cyphermox)]

After verification this particular fix (for appmenu-gtk, which directly affects nm-applet leaking) is already present in quantal and raring, so we're only targetting precise... I'm marking that task as In Progress and I'll upload the fix from Alex shortly.

[Mathieu Trudel-Lapierre (cyphermox)]

Urgh, that's already tracked in bug 787736; no need to track it here.

[Chris Halse Rogers (raof)]

Hello Harm, or anyone else affected,

Accepted network-manager-applet into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/network-manager-applet/0.9.4.1-0ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Martin Spacek (mspacek)]

What's the status of this bug for quantal? Surely if there's a fix for precise, there should be one for quantal, no? In Xubuntu 12.10, after a day of running on WiFi, my submenus still show up as empty and the network shows as disconnected, even though I'm still connected to the same access point and networking still works. Also, more recently I've noticed that the nm-applet status icon in the indicator plugin stops showing signal levels around the same time, or maybe a few days later (I'm using the dark Blackbird theme).

[Roman Yepishev (rye)]

I experience exactly the same symptoms as the bug report description has.

After some time nm-applet stops reacting to user input and submenu entries do not populate. This is with latest 12.10 packages.

I noticed that the indicator tries to get the menu but it fails - dbus-monitor output below:
method call sender=:1.12 -> dest=:1.67 serial=14296 path=/com/canonical/Unity/Panel/Service; interface=com.canonical.Unity.Panel.Service; member=ShowEntry
   string "0x26ef470"
   uint32 0
   int32 1147
   int32 24
   uint32 1
   uint32 1356027813
method return sender=:1.67 -> dest=:1.12 reply_serial=14296
method call sender=:1.67 -> dest=:1.14 serial=107621 path=/org/ayatana/NotificationItem/nm_applet/Menu; interface=com.canonical.dbusmenu; member=AboutToShowGroup
   array [
      int32 80799
   ]
error sender=:1.14 -> dest=:1.67 error_name=org.gtk.GDBus.UnmappedGError.Quark._LIBDBUSMENU_2dGLIB.Code0 reply_serial=107621
   string "The IDs supplied '[80799]' do not refer to any menu items we have"

When some item is clicked (e.g. disconnect for current wifi AP) the following happens:
signal sender=:1.67 -> dest=(null destination) serial=107704 path=/com/canonical/Unity/Panel/Service; interface=com.canonical.Unity.Panel.Service; member=EntryActivated
   string ""
   struct {
      int32 0
      int32 0
      uint32 0
      uint32 0
   }
method call sender=:1.67 -> dest=:1.14 serial=107705 path=/org/ayatana/NotificationItem/nm_applet/Menu; interface=com.canonical.dbusmenu; member=EventGroup
   array [
      struct {
         int32 80846
         string "clicked"
         variant int32 0
         uint32 106251172
      }
   ]
error sender=:1.14 -> dest=:1.67 error_name=org.gtk.GDBus.UnmappedGError.Quark._LIBDBUSMENU_2dGLIB.Code0 reply_serial=107705
   string "The IDs supplied '[80846]' do not refer to any menu items we have"

So this looks like like dbus menu is losing the IDs or nm-applet is using the ids that are no longer valid.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Harm van Bakel (hvbakel)]

I have been testing the network-manager-gnome package in the precise-proposed repository (0.9.4.1-0ubuntu2.1) for a few weeks now, but unfortunately the applet menu still stopped working on several occasions. Unfortunately it looks like the problem is still not completely fixed with this latest update, though (subjectively) it now seems to take longer to occur.

[Harm van Bakel (hvbakel)]

I should add that I haven't experienced any memory leaks since upgrading to Precise, the only problem I'm still experiencing is that the app menu typically stops working within 1 or more days after a fresh boot.

[Cody A.W. Somerville (cody-somerville)]

This seems to occur frequently after suspend/resume. I see the same errors as reported in comment #69.

The errors are returned by bus_event_group or bus_about_to_show_group in libdbusmenu-glib/server.c.

As such, I don't think the memory leaks are related to the issue with nm not responding / not showing submenus.

[James M. Leddy (jm-leddy)]

moving to fixed released in light of comment #71

[Eloy Paris (peloy-chapus)]

Is this bug about a memory leak, about the applet stopping to work, or both? If it is about the applet stopping to work after a while then this bug is not fixed -- the issue still exists in 12.10.

Just wondering about what exactly is being claimed is fixed.

[Hadrien Titeux (hadware)]

Definitely! I updated to 12.10 on a falty PC (one that suffered this bug) and it didn't fix anything! This isn't fixed at all!

[Teo (teo1978)]

I guess this is about both things because the memory leak was thought to be the cause of the stopping to work. I guess it was claimed as fixed because the memory leak probably _was_ fixed (that's what other said, I have no idea) but apparently it truned out to not be the cause of the stopping to work thing.

There was a bug report that perfectly describes the issue that is still not fixed,
https://bugs.launchpad.net/ubuntu/+source/network-manager-applet/+bug/930563
but it was marked as a duplicate of this one.

[Harm van Bakel (hvbakel)]

@matteosistisette: Indeed, I initially thought the two problems were related, hence the title. In retrospect it would have been better if I had submitted two separate bugs to avoid this confusion.

[Neal McBurnett (nealmcb)]

Based on the comments both here and in bug 930563 that nm-applet still stops functioning after a while in quantal 12.10 despite a fix for the memory leak problem, I marked that as no longer a "duplicate" of this one. So sign up there if this exists for you, and let's focus here on the memory leak.

[Cody A.W. Somerville (cody-somerville)]

I spoke with Mathieu Trudel-Lapierre, who is the developer assigned to this bug, and he has clarified that this bug is indeed for the nm-applet becoming unresponsive to interactions and submenus showing an empty list issue. The fix for the leaks is correct but it did not fix the underlaying issue that is causing the behaviour described in the bug description. I think James M. Leddy misspoke/typoed in his comment - as you can see, he actually changed the statuses from 'Fix Released' and 'Fix Committed' to 'Confirmed'.

Neal: Can you please remark bug 930563 as a duplicate of this bug?

[Neal McBurnett (nealmcb)]

Cody, thanks for looking more into it. It seems like I might have just piled on to the confusion over the scope of the bug etc., though the confusion has indeed been here a long time. I don't object to the other bug being changed back to being a duplicate if it is still being worked on, especially given that this one has a higher priority. But I'll leave it to someone else who is more confident in how to handle all this to do the actual re-dup, and hopefully to clarify the titles, descriptions, etc.
And James, can you clarify why under OEM Priority Project this one is still listed as "Fix committed" for Quantal?
Thanks, all.

[James M. Leddy (jm-leddy)]

If we're using this bug for the wider submenu problem then we should back out the SRU so that we can allow other srus such as bug 445872 to proceed.

[Mathieu Trudel-Lapierre (cyphermox)]

Well, in light that there actually were memory leaks and that the actual update is indeed beneficial, I'd actually rather we mark that as Fix Released and verification-done --

It will remain that there are still dbusmenu issues (or problems somewhere else in the stack nm-applet uses) that cause the menus to become unresponsive, as shown by the dbus traces above.

[Adrian Bridgett (adrian-bridgett)]

It's great that memory leaks have been fixed (a much under-appreciated problem), however that said marking it as "fixed" when the actually menu disappearing isn't fixed doesn't seem quite right to me. The restart workaround is a PITA for technical users and just not going to happen with non-technical users.

[Mathieu Trudel-Lapierre (cyphermox)]

Adrian, the underlying bug is still going to be fixed; but we need to separate fixing something in nm-applet and fixing something in another package.

The leaks are fixed in Raring, marking as Fix Released for that release.

[Adrian Bridgett (adrian-bridgett)]

Ah, thanks for the explanation Mathieu - I thought it was a bug in nm-applet's dbus code :-)

[James M. Leddy (jm-leddy)]

I've broken this into bug 1117730. Anyone interested in following the fix for the menu problem should go to that bug. This bug is now refocused to fix the memory leak described in comment #67. We need to move these SRUs to verificaton because they're blocking all remaining SRUs for network manager, and this was the bug referenced in the changelog.

[Brian Murray (brian-murray)]

Before releasing this update it'd be worthwhile to investigate the following crash reported on errors.ubuntu.com with only the version from network-manager-applet in precise-proposed.

https://errors.ubuntu.com/bucket/?id=failed%3A%2Fusr%2Fbin%2Fnm-connection-editor%3A11%3Ax86_64%3A%2Fusr%2Flib%2Flibnm-glib.so.4.3.0%2B1e884%3A%2Fusr%2Flib%2Flibnm-glib.so.4.3.0%2B1ea3c%3A%2Flib%2Fx86_64-linux-gnu%2Flibdbus-1.so.3.5.8%2Bb59a%3A%2Flib%2Fx86_64-linux-gnu%2Flibdbus-1.so.3.5.8%2Be82a%3A%2Fusr%2Flib%2Fx86_64-linux-gnu%2Flibdbus-glib-1.so.2.2.2%2Ba425%3A%2Flib%2Fx86_64-linux-gnu%2Flibglib-2.0.so.0.3200.3%2B47d53%3A%2Flib%2Fx86_64-linux-gnu%2Flibglib-2.0.so.0.3200.3%2B480a0%3A%2Flib%2Fx86_64-linux-gnu%2Flibglib-2.0.so.0.3200.3%2B4849a%3A%2Fusr%2Fbin%2Fnm-connection-editor%2B110c3

This has occurred 57 times as of today. Its possible that there may be a corresponding crash that has not failed retracing (notice the failed in the url) which may have affected previous versions of nm-applet.

[Brian Murray (brian-murray)]

To clarify this error only appears with the version of the package from precise-proposed.

[James M. Leddy (jm-leddy)]

I've been running on precise for a few months now. Similar to Alex's comment #57 I was not able to observe the problem after running valgrind for a few hours. I consider this done from a verification standpoint.

[Launchpad Janitor (janitor)]

This bug was fixed in the package network-manager-applet - 0.9.4.1-0ubuntu2.1

---------------
network-manager-applet (0.9.4.1-0ubuntu2.1) precise-proposed; urgency=low

  * Backport r355, r364, r368 from trunk (LP: #780602)
    - debian/patches/nm-applet-use-indicator.patch: Plug two small leaks.
    - debian/patches/git_fix_some_leaks_80ef61b.patch: cherry-picked patch to
      fix a few leaks: g_object_get() and gtk_tree_model_get() copy/ref the
      values they return, so make sure to deal with that everywhere.
    - debian/patches/git_mac_addr_string_leakage_6dae878.patch: use
      nm_utils_hwaddr_ntoa() to output MAC addresses as strings since it's
      available and make sure they get freed to avoid leaks.
 -- Alex Chiang <email address hidden> Fri, 23 Nov 2012 13:30:37 -0500

[Brian Murray (brian-murray)]

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "appmenu-gtk.debdiff" of this bug report has been identified as being a patch in the form of a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors team please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Dimitri John Ledkov (xnox)]

bug #787736 already is handling appmenu-gtk memory fix backport, so in fact the patch "appmenu-gtk.debdiff" is already uploaded with bug #787736 as reference. Unsubscribing sponsors.

[Mathieu Trudel-Lapierre (cyphermox)]

https://launchpad.net/~mathieu-tl/+archive/sru-staging/+packages

On the above PPA I've made available has updated packages for Precise, Quantal and Raring to help debugging the menu functionality. *it does not fix the issue*

It would be helpful if people could try these packages and help debugging the issue by providing more information, basically, after updating the pacakge, kill off nm-applet and start the new one like so in a terminal:

G_MESSAGES_DEBUG=nm-applet-indicator nm-applet

It will start outputting debug information on the console, we'll want to match that output with the output from dbus-monitor in another terminal:

dbus-monitor --session --monitor interface=com.canonical.dbusmenu

Then we'll match all of that to help figure out how to fix this bug.

[Marlon Nelson (marlon-nelson)]

Here's the output from:

dbus-monitor --session --monitor interface=com.canonical.dbusmenu 2>&1 | tee dbus-monitor.out

[Marlon Nelson (marlon-nelson)]

and the output from:

G_MESSAGES_DEBUG=nm-applet-indicator nm-applet 2>&1 | tee nm-applet.out

[Eloy Paris (peloy-chapus)]

Output from G_MESSAGES_DEBUG=nm-applet-indicator nm-applet | tee nm-applet-debug.txt.

Stopped running after the menus in the applet became non-functional. This took several hours. I saw many occurrences of lines like:

(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36070
(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36071
(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36072
(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36073
(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36074
(nm-applet:4661): nm-applet-indicator-DEBUG: Just set up menu for ID 36093

Over and over. Does this mean that menus are getting created with nothing that destroys them?

Thanks for looking into this critical bug.

[Eloy Paris (peloy-chapus)]

Output from "dbus-monitor --session --monitor interface=com.canonical.dbusmenu | tee dbus-monitor-output.txt" that corresponds to the nm-applet output in the previous comment, i.e. taken at the same time. Also stopped after a few hours when the nm-applet menus stopped working.

[Alex Chiang (achiang)]

dbus attachment

[Alex Chiang (achiang)]

nm-applet dbug attached

Both of my files were obtained on precise.

$ dpkg -l | grep mtrudel
ii libnm-gtk-common 0.9.4.1-0ubuntu2.2~srustaging~mtrudel1 network management framework (common files for wifi and mobile)
ii libnm-gtk0 0.9.4.1-0ubuntu2.2~srustaging~mtrudel1 network management framework (GNOME dialogs for wifi and mobile)
ii network-manager-gnome 0.9.4.1-0ubuntu2.2~srustaging~mtrudel1 network management framework (GNOME frontend)
ii network-manager-gnome-dbg 0.9.4.1-0ubuntu2.2~srustaging~mtrudel1 network management framework (debugging symbols)

$ dpkg -l | grep appmenu-gtk
ii appmenu-gtk 0.3.92-0ubuntu1.1 Export GTK menus over DBus
ii appmenu-gtk3 0.3.92-0ubuntu1.2 Export GTK menus over DBus

[Scott Talbert (swt-techie)]

@Mathieu - have these debug logs provided any useful information for resolving this bug, or do you still need additional logs?

[Mathieu Trudel-Lapierre (cyphermox)]

We don't need so much additional information anymore from the debug packages. Yes, the information was useful.

I was able to write a small python script that reproduces the issue:
https://code.launchpad.net/~mathieu-tl/+junk/test-indicator-update

Playing with the delays there and the number of updates in a row, you can easily wedge dbusmenu in a matter of about an hour.

As far as I'm concerned, this proves that the issue isn't in nm-applet but somewhere in the way that dbusmenu takes care of updates to the menus, but I'm still not quite sure what is wong with it.

Any help is of course welcome :)

[Mathieu Trudel-Lapierre (cyphermox)]

Let's please move discussions about the menus being empty to bug 1117730.

[Jamie Boyle (jpwboyle)]

I've had this problem of visible but unresponsive nm-applet in 12.10 and 13.04 (upgraded about a week ago), since removing Windows for 12.10 maybe a year ago. I think this is pretty critical - any "normal" person would restart the computer every time, so if there's anything not-too-complex I can do to help, let me know.

I have WiFi and a 3G card in my Lenovo X220T laptop and spend a lot of time in cafes using public WiFi, but when the WiFi is bad, I change to 3G. I haven't noticed a pattern about what causes the unresponsiveness.

[James M. Leddy (jm-leddy)]

On 06/26/2013 07:58 AM, Jamie Boyle wrote:
> I've had this problem of visible but unresponsive nm-applet in 12.10 and
> 13.04 (upgraded about a week ago), since removing Windows for 12.10
> maybe a year ago. I think this is pretty critical - any "normal" person
> would restart the computer every time, so if there's anything not-too-
> complex I can do to help, let me know.
>
> I have WiFi and a 3G card in my Lenovo X220T laptop and spend a lot of
> time in cafes using public WiFi, but when the WiFi is bad, I change to
> 3G. I haven't noticed a pattern about what causes the unresponsiveness.
>
Please see Bug 1011073. The problem is fixed in -proposed

[James M. Leddy (jm-leddy)]

Please refer to to bug 1011073.