suspected memory leak with indicator-keyboard (causing gnome-session-flashback to freeze after startup)

[prusswan]

[ Impact ]

 * The indicator-keyboard-service process suffers from a memory leak that quickly consumes the entire available memory of the system until the process is killed by the OOM daemon. Since indicator-keyboard is a session service it will be restarted and the process repeats, making the system unusable. On the system where I tested this indicator-keyboard-service started with ~2MB and quickly rose to consume all 64GB of memory. The session then appears to hang for about a minute until the OOM daemon kills the process.

[ Test Plan ]

 * Prerequisites:
    - A running gnome-session-flashback XOrg session

 * Ensure you have multiple input sources configured, for example:
   $ dconf read /org/gnome/desktop/input-sources/sources
   [('xkb', 'us'), ('ibus', 'libpinyin')]

 * Start indicator-keyboard-service via:
   $ indicator-keyboard-service --use-gtk

 * Watch the memory consumption to raise quickly and consume all available memory until it is being killed by the OOM daemon.

 * Expected base case:
    - After starting the indicator-keyboard-service the language indicator appears in the indicator-applet widget and displays the active language.
   - Clicking on the indicator-keyboard icon in the applet shows a menu and allows switching between languages.

[ Where problems could occur ]

 * The regression potential is very low since indicator-keyboard-service is already suffering from a memory leak that is causing it to be killed by the OOM daemon every few minutes. It is unlikely that this change will make the problem worse.

 * User base of indicator-keyboard is low. The main users of this package are Unity and Gnome Flashback and both are not default desktops on Ubuntu. Gnome Shell does not depend on indicator-keyboard. So a potential regression would only affect users of these desktops.

 * The source code changes include the removal of a redundant initialization of IBus in two functions of indicator-keyboard-service and instead move the initialization to the constructor of the indicator keyboard service. No new functionality is added and the existing behavior of the application is not changes except for consolidation of the redundant IBus initialization calls.

 * On GNOME Flashback another workaround exists in case there is an unexpected regression due to this chaneg. In the unlikely event that the change has a side effect and prevents the indicator from working, users an switch to the newer "System Indicators" applet and remove the indicator-keyboard package completely.

NEWS

The fix is available in version 0.0.0+19.10.20240924-0ubuntu1

=== Original Description ===

Description:

The system was previously on 22.04, then upgraded to 23.10 to the current daily 24.04. System will freeze up a few minutes after startup (with no apparent related crashes). After various trial and error, the possible cause is isolated to indicator-keyboard when running gnome-session-flashback (freeze does not happen anymore with flashback interface, after indicator-keyboard package is removed). Not sure why this is not happening with vanilla gdm3. Also, the freeze was first observed after the upgrade to 23.10.

**logs below taken from `sudo journalctl --since "3 hours ago" | grep indicator`**

Feb 29 16:49:03 machine kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[e-mail]/app.slice/indicator-keyboard.service,task=indicator-keybo,pid=5810,uid=1000
Feb 29 16:49:03 machine kernel: Out of memory: Killed process 5810 (indicator-keybo) total-vm:66620172kB, anon-rss:64296784kB, file-rss:768kB, shmem-rss:0kB, UID:1000 pgtables:129412kB oom_score_adj:200
Feb 29 16:49:03 machine kernel: oom_reaper: reaped process 5810 (indicator-keybo), now anon-rss:688kB, file-rss:768kB, shmem-rss:0kB
Feb 29 16:48:56 machine systemd[5304]: indicator-keyboard.service: A process of this unit has been killed by the OOM killer.
Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service: Main process exited, code=killed, status=9/KILL
Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service: Failed with result 'oom-kill'.
Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service: Consumed 1min 11.913s CPU time.
Feb 29 16:48:58 machine systemd[5304]: indicator-keyboard.service: Scheduled restart job, restart counter is at 1.
Feb 29 16:48:58 machine systemd[5304]: Started indicator-keyboard.service - Indicator Keyboard Backend.
Feb 29 17:00:40 machine dbus-daemon[1111]: [system] Activating via systemd: service name='org.freedesktop.login1' unit='dbus-org.freedesktop.login1.service' requested by ':1.9' (uid=0 pid=1588 comm="apt remove indicator-keyboard" label="unconfined")

[aein (aeinner)]

I am experiencing similar problem with 24.04 after adding ibus-mozc method. Right after login indicator-keyboard starts using 100% of CPU with rapid memory consumption which results in freezing and OOM. Symptomatic fix: removing --use-gtk solves this problem.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Chris Guiver (guiverc)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command only once, as it will automatically gather debugging information, in a terminal:

apport-collect 2055388

When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

[Valentin Lab (vaab)]

Same here. Went from an Ubuntu 22.04 to a 24.04.1 (using unity desktop on X). At login time, everything was slow, and my 64Gb of ram were swallowed by `indicator-keyboard-service` according to `top` (with `M` shortcut to sort by memory taken).

Had to `sudo mv /usr/libexec/indicator-keyboard/indicator-keyboard-service{,.old}` before killing it (as it will be restarted by `systemctl`), then `killall -9 indicator-keyboard-service`... Did this before taking the time to investigate properly.

I confirm the following work around (as suggested by @aein):

`systemctl --user edit indicator-keyboard.service` to add an override with:

```
[Service]
ExecStart=
ExecStart=/usr/libexec/indicator-keyboard/indicator-keyboard-service
```

(thus removing the `--use-gtk` as @aein mentionned), saving the file, quitting. Followed by:

```
## Only if necessary (if the move was done as I did)
sudo mv /usr/libexec/indicator-keyboard/indicator-keyboard-service{.old,}
```

Then:
```
systemctl --user daemon-reload
systemctl --user restart indicator-keyboard.service
```

Fixed everything for me.

[prusswan (prusswan)]

Chris, do you mean to execute apport-collect while the problem is
happening? Bearing in mind this issue renders the system unusable within
minutes upon startup?

On Fri, Aug 30, 2024 at 5:15 PM Chris Guiver <email address hidden>
wrote:

> Thank you for taking the time to report this bug and helping to make
> Ubuntu better. Please execute the following command only once, as it
> will automatically gather debugging information, in a terminal:
>
> apport-collect 2055388
>
> When reporting bugs in the future please use apport by using 'ubuntu-
> bug' and the name of the package affected. You can learn more about this
> functionality at https://wiki.ubuntu.com/ReportingBugs.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2055388
>
> Title:
> suspected memory leak with indicator-keyboard (causing gnome-session-
> flashback to freeze after startup)
>
> Status in indicator-keyboard package in Ubuntu:
> Confirmed
>
> Bug description:
> Description:
>
> The system was previously on 22.04, then upgraded to 23.10 to the
> current daily 24.04. System will freeze up a few minutes after startup
> (with no apparent related crashes). After various trial and error, the
> possible cause is isolated to indicator-keyboard when running gnome-
> session-flashback (freeze does not happen anymore with flashback
> interface, after indicator-keyboard package is removed). Not sure why
> this is not happening with vanilla gdm3. Also, the freeze was first
> observed after the upgrade to 23.10.
>
> **logs below taken from `sudo journalctl --since "3 hours ago" | grep
> indicator`**
>
> Feb 29 16:49:03 machine kernel:
> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=user.slice,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/[e-mail]
> /app.slice/indicator-keyboard.service,task=indicator-keybo,pid=5810,uid=1000
> Feb 29 16:49:03 machine kernel: Out of memory: Killed process 5810
> (indicator-keybo) total-vm:66620172kB, anon-rss:64296784kB, file-rss:768kB,
> shmem-rss:0kB, UID:1000 pgtables:129412kB oom_score_adj:200
> Feb 29 16:49:03 machine kernel: oom_reaper: reaped process 5810
> (indicator-keybo), now anon-rss:688kB, file-rss:768kB, shmem-rss:0kB
> Feb 29 16:48:56 machine systemd[5304]: indicator-keyboard.service: A
> process of this unit has been killed by the OOM killer.
> Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service: Main
> process exited, code=killed, status=9/KILL
> Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service:
> Failed with result 'oom-kill'.
> Feb 29 16:48:57 machine systemd[5304]: indicator-keyboard.service:
> Consumed 1min 11.913s CPU time.
> Feb 29 16:48:58 machine systemd[5304]: indicator-keyboard.service:
> Scheduled restart job, restart counter is at 1.
> Feb 29 16:48:58 machine systemd[5304]: Started
> indicator-keyboard.service - Indicator Keyboard Backend.
> Feb 29 17:00:40 machine dbus-daemon[1111]: [system] Activating via
> systemd: service name='org.freedesktop.login1'
> unit='dbus-org.freedesktop.login1.service' requested by ':1.9' (uid=0
> ...

[Sebastian Geiger (lanoxx)]

I debugged this today and found the following. When we attach a debugger we very quickly end-up in g_resource_new_from_data:

(gdb) backtrace
#0 0x0000792a248ad645 in __GI___libc_malloc (bytes=bytes@entry=80) at ./malloc/malloc.c:3301
#1 0x0000792a24b1fafa in g_malloc (n_bytes=80) at ../../../glib/gmem.c:100
#2 0x0000792a24da5272 in gvdb_table_new_from_bytes (bytes=0x5fe9aca33360, trusted=1, error=0x7fff0e5cd9f0) at ../../../subprojects/gvdb/gvdb/gvdb-reader.c:147
#3 0x0000792a24cfd30a in g_resource_new_from_data (data=data@entry=0x5fe9aca33360, error=error@entry=0x0) at ../../../gio/gresource.c:627
#4 0x0000792a24cfd3f9 in register_lazy_static_resources_unlocked () at ../../../gio/gresource.c:1398
#5 0x0000792a24cfd743 in register_lazy_static_resources () at ../../../gio/gresource.c:1417
#6 register_lazy_static_resources () at ../../../gio/gresource.c:1411
#7 g_resources_enumerate_children (path=path@entry=0x5fd5afe8dad0 "/org/gtk/libgtk/icons/16x16/actions/", lookup_flags=lookup_flags@entry=G_RESOURCE_LOOKUP_FLAGS_NONE, error=error@entry=0x0) at ../../../gio/gresource.c:1250
#8 0x0000792a251c4514 in scan_resources (icon_theme=<optimized out>, full_dir=0x5fd5afe8dad0 "/org/gtk/libgtk/icons/16x16/actions/", dir=0x5fd5afd30af0) at ../../../gtk/gtkicontheme.c:3258
#9 theme_subdir_load (icon_theme=icon_theme@entry=0x5fd5afacf9e0, theme=theme@entry=0x5fd5afd2f410, theme_file=theme_file@entry=0x5fd5afe4d620, subdir=0x5fd5afd2f460 "16x16/actions") at ../../../gtk/gtkicontheme.c:3439
#10 0x0000792a251c4b56 in insert_theme (icon_theme=icon_theme@entry=0x5fd5afacf9e0, theme_name=<optimized out>) at ../../../gtk/gtkicontheme.c:1239
#11 0x0000792a251c4bed in insert_theme (icon_theme=icon_theme@entry=0x5fd5afacf9e0, theme_name=<optimized out>) at ../../../gtk/gtkicontheme.c:1259
#12 0x0000792a251c4bed in insert_theme (icon_theme=icon_theme@entry=0x5fd5afacf9e0, theme_name=<optimized out>) at ../../../gtk/gtkicontheme.c:1259
#13 0x0000792a251c4bed in insert_theme (icon_theme=icon_theme@entry=0x5fd5afacf9e0, theme_name=<optimized out>) at ../../../gtk/gtkicontheme.c:1259
#14 0x0000792a251c4d30 in load_themes (icon_theme=0x5fd5afacf9e0) at ../../../gtk/gtkicontheme.c:1373
#15 ensure_valid_themes (icon_theme=icon_theme@entry=0x5fd5afacf9e0) at ../../../gtk/gtkicontheme.c:1482
#16 0x0000792a251c6037 in real_choose_icon (icon_theme=icon_theme@entry=0x5fd5afacf9e0, icon_names=icon_names@entry=0x7fff0e5ce130, size=size@entry=22, scale=scale@entry=1, flags=0) at ../../../gtk/gtkicontheme.c:1696
#17 0x0000792a251c67ac in choose_icon (icon_theme=icon_theme@entry=0x5fd5afacf9e0, icon_names=icon_names@entry=0x7fff0e5ce130, size=size@entry=22, scale=scale@entry=1, flags=flags@entry=0) at ../../../gtk/gtkicontheme.c:2015
#18 0x0000792a251c6c51 in gtk_icon_theme_lookup_icon_for_scale (icon_theme=icon_theme@entry=0x5fd5afacf9e0, icon_name=icon_name@entry=0x5fd5afbdc2c0 "indicator-keyboard-En", size=size@entry=22, scale=scale@entry=1, flags=flags@entry=0) at ../../../gtk/gtkicontheme.c:2161
#19 0x0000792a251c6dbd in gtk_icon_theme_lookup_icon (icon_theme=icon_theme@entry=0x5fd5afacf9e0, icon_name=icon_name@entry=0x5fd5afbdc2c0 "indicator-keyboard-En", size=size@...

[Sebastian Geiger (lanoxx)]

Fixed in https://bazaar.launchpad.net/~indicator-applet-developers/indicator-keyboard/trunk/revision/2213

[Launchpad Janitor (janitor)]

This bug was fixed in the package indicator-keyboard - 0.0.0+19.10.20240924-0ubuntu1

---------------
indicator-keyboard (0.0.0+19.10.20240924-0ubuntu1) oracular; urgency=medium

  [ Sebastian Geiger ]
  * Fix memory leak in gresource (LP: #1384384, #2055388).

 -- Dmitry Shachnev <email address hidden> Tue, 24 Sep 2024 23:52:04 +0300

[Alberts Muktupāvels (muktupavels)]

Related fix in ibus https://github.com/ibus/ibus/commit/9f50b46f82024efa59dd1d35a3d89c4350cd1cff

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robie Basak (racb)]

> Test Plan

Please could you amend this to verify that indicator-keyboard still works correctly in a basic use case?

[Robie Basak (racb)]

Hello prusswan, or anyone else affected,

Accepted indicator-keyboard into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/indicator-keyboard/0.0.0+19.10.20220803-0ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[prusswan (prusswan)]

Have installed the indicator-keyboard from proposed. System hasn't frozen yet (a good sign)

$ sudo apt install indicator-keyboard/noble-proposed

$ sudo apt policy indicator-keyboard
indicator-keyboard:
  Installed: 0.0.0+19.10.20220803-0ubuntu3.1
  Candidate: 0.0.0+19.10.20220803-0ubuntu3.1
  Version table:
 *** 0.0.0+19.10.20220803-0ubuntu3.1 100
        100 http://archive.ubuntu.com/ubuntu noble-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     0.0.0+19.10.20220803-0ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages

Update:

While the system no longer freezes, the indicator-keyboard only show the current input language (e.g. "EN") even if other input options are present, so not able to switch inputs through the applet UI (but switching via keyboard shortcut still works).

[Andreas Hasenack (ahasenack)]

> While the system no longer freezes, the indicator-keyboard only show the current input language (e.g.
> "EN") even if other input options are present, so not able to switch inputs through the applet UI (but
> switching via keyboard shortcut still works).

Is this a regression? Were you able to switch to other inputs using the UI before?

[prusswan (prusswan)]

> Is this a regression? Were you able to switch to other inputs using the UI before?

I believe so (able to switch inputs), but due to the OOM issue I have not been using indicator-keyboard after upgrading from 22.04. Even if this is a regression, it might not be related to the issue or the proposed fix. Perhaps another user can confirm?

[Sebastian Geiger (lanoxx)]

I have updated the description and added the expected base case.

I have installed version 0.0.0+19.10.20220803-0ubuntu3.1 of indicator-keyboard. I can confirm that the memory is stable and using ~4MB of memory.

When I start it with systemctl --user indicator-keyboard.service, then in the indicator-keyboard applet the language icon appears. So this fix makes the applet usable again. I have done the following tests:

- Check if I can use CTRL+Space to switch languages and observe that the indicator-keyboard applet icon updates itself to show the active language (works; also works for CN)
- Check if I can use the menu icon from the indicator-keyboard applet to switch languages (works)

Independently from the above, I can confirm the observation reported by prusswan. I was able to check with Alberts, that this is unrelated to this bug and this regression already existed in version 0.0.0+19.10.20220803-0ubuntu3 and thus it should not block verification of this bug.

The behavior that prusswan observes, seems to be that some languages are missing from the indicator-keyboard applet. For me it is as following. I have three languages configured (EN, DE, CN:pinyin) on my system. In the system-indicators applet (which is an unrelated applet) I can see all three languages and I can select the Chinese input from the menu and it gets correctly activated (same for changing with CTRL+Space.

However, in the indicator-keyboard applet menu I only see EN and DE (CN is missing). Since this behavior was already present in 0.0.0+19.10.20220803-0ubuntu3 it should not block us from accepting this bug fix.

[Launchpad Janitor (janitor)]

This bug was fixed in the package indicator-keyboard - 0.0.0+19.10.20220803-0ubuntu3.1

---------------
indicator-keyboard (0.0.0+19.10.20220803-0ubuntu3.1) noble; urgency=medium

  [ Sebastian Geiger ]
  * Fix memory leak in gresource (LP: #2055388).

 -- Dmitry Shachnev <email address hidden> Tue, 24 Sep 2024 23:52:04 +0300

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for indicator-keyboard has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.