incredibly high memory usage

[beadon]

Please describe the issue you’re experiencing:
gnome-software is consuming 6+ GB of RAM on my machine. This is unacceptable and is affecting all other applications.
Error message appearing in /var/log/syslog
N/A -- other processes are fighting for memory, and OOM killer is more active.
What Linux distribution are you using, and what kind of package (RPM, deb, flatpak, etc.) are you experiencing the problem with?
Ubuntu 22.10
apt
flatpak

I don't know how to replicate this issue, however it does appear to be a problem retaining memory. See LSOF report below, software is gnome-software 43.0

```
sudo lsof -p 750448 -n > ~/ramusage.txt
cat ramusage.txt | perl -lane 'print "$F[6] $F[0] $F[1] $F[2] $F[3] $F[4] $F[5] $F[7] $F[8] $F[9] $F[10]"' | sort -rn > sorted-report.txt
```

Top 10 filehandles are holding onto something like 6GB of RAM?! What's going on here ?
(open filehandles, sorted by bytes):

```
  1 157191688 gnome-sof 750448 beadon 11u netlink 0t0 ROUTE
  2 103546882 gnome-sof 750448 beadon DEL REG 259,2 /usr/share/icons/hicolor/icon-theme.cache
  3 103022860 gnome-sof 750448 beadon DEL REG 259,2 /usr/share/glib-2.0/schemas/gschemas.compiled
  4 61210892 gnome-sof 750448 beadon DEL REG 259,2 /var/lib/flatpak/exports/share/icons/hicolor/icon-theme.cache
  5 60828700 gnome-sof 750448 beadon DEL REG 259,2 /var/cache/fontconfig/2bd0278d-49c2-4c8c-9eea-32d695b22756-le64.cache-7
  6 30430848 gnome-sof 750448 beadon mem REG 259,2 101455545 /usr/lib/x86_64-linux-gnu/libicudata.so.71.1
  7 14516125 gnome-sof 750448 beadon mem REG 259,2 97416011 /home/beadon/.cache/gnome-software/flatpak-system-default/components.xmlb
  8 11308392 gnome-sof 750448 beadon mem REG 259,2 101450031 /usr/lib/x86_64-linux-gnu/librsvg-2.so.2.48.0
  9 8254320 gnome-sof 750448 beadon mem REG 259,2 101450280 /usr/lib/x86_64-linux-gnu/libgtk-4.so.1.800.1
 10 7599984 gnome-sof 750448 beadon 475u REG 259,2 60821438 /var/tmp/#60821438 (deleted)
 11 7582821 gnome-sof 750448 beadon 476u REG 259,2 60821485 /var/tmp/#60821485 (deleted)
 12 6733824 gnome-sof 750448 beadon 478u REG 259,2 60821699 /var/tmp/#60821699 (deleted)
 13 5966212 gnome-sof 750448 beadon mem REG 259,2 97419097 /home/beadon/.cache/gnome-software/appstream/components.xmlb
 14 5712208 gnome-sof 750448 beadon mem REG 259,2 101450114 /usr/lib/locale/locale-archive
 15 5681431 gnome-sof 750448 beadon 477u REG 259,2 60821584 /var/tmp/#60821584 (deleted)
 16 5469177 gnome-sof 750448 beadon 482u REG 259,2 60817847 /var/tmp/#60817847 (deleted)
 17 5406720 gnome-sof 750448 beadon 479u REG 259,2 60821593 /var/tmp/#60821593 (deleted)
 18 5341166 gnome-sof 750448 beadon 493u REG 259,2 60821670 /var/tmp/#60821670 (deleted)
 19 4616704 gnome-sof 750448 beadon 206u REG 259,2 60819152 /var/tmp/#60819152 (deleted)
 20 4519958 gnome-sof 750448 beadon 497u REG 259,2 60821824 /var/tmp/#60821824 (deleted)
```

Details:
[sorted-report.txt](/uploads/43c17fb8547366c33c3e49931a5a07b7/sorted-report.txt)

ProblemType: Bug
DistroRelease: Ubuntu 22.10
Package: xorg 1:7.7+23ubuntu2
ProcVersionSignature: Ubuntu 5.19.0-29.30-generic 5.19.17
Uname: Linux 5.19.0-29-generic x86_64
ApportVersion: 2.23.1-0ubuntu3
Architecture: amd64
BootLog:

CasperMD5CheckResult: pass
CompositorRunning: None
CurrentDesktop: ubuntu:GNOME
Date: Sun Feb 5 20:06:15 2023
DistUpgraded: 2023-01-14 21:42:30,924 DEBUG Running PostInstallScript: '/usr/lib/ubuntu-advantage/upgrade_lts_contract.py'
DistroCodename: kinetic
DistroVariant: ubuntu
DkmsStatus:
 8812au/5.6.4.2_35491.20191025, 5.15.0-58-generic, x86_64: installed
 8812au/5.6.4.2_35491.20191025, 5.19.0-29-generic, x86_64: installed
 nvidia/525.78.01, 5.19.0-29-generic, x86_64: installed
 virtualbox/6.1.38, 5.15.0-58-generic, x86_64: installed
 virtualbox/6.1.38, 5.19.0-29-generic, x86_64: installed
ExtraDebuggingInterest: Yes, including running git bisection searches
GraphicsCard:
 Intel Corporation TigerLake-LP GT2 [Iris Xe Graphics] [8086:9a49] (rev 01) (prog-if 00 [VGA controller])
   Subsystem: Lenovo TigerLake-LP GT2 [Iris Xe Graphics] [17aa:22d4]
InstallationDate: Installed on 2023-01-09 (27 days ago)
InstallationMedia: Ubuntu 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809.1)
MachineType: LENOVO 20XY0027US
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.19.0-29-generic root=UUID=1cae8af8-977f-4853-9106-9169f34c4bc2 ro quiet splash vt.handoff=7
SourcePackage: xorg
Symptom: display
UpgradeStatus: Upgraded to kinetic on 2023-01-15 (21 days ago)
dmi.bios.date: 12/05/2022
dmi.bios.release: 1.58
dmi.bios.vendor: LENOVO
dmi.bios.version: N32ET82W (1.58 )
dmi.board.asset.tag: Not Available
dmi.board.name: 20XY0027US
dmi.board.vendor: LENOVO
dmi.board.version: SDK0J40697 WIN
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 31
dmi.chassis.vendor: LENOVO
dmi.chassis.version: None
dmi.ec.firmware.release: 1.33
dmi.modalias: dmi:bvnLENOVO:bvrN32ET82W(1.58):bd12/05/2022:br1.58:efr1.33:svnLENOVO:pn20XY0027US:pvrThinkPadX1YogaGen6:rvnLENOVO:rn20XY0027US:rvrSDK0J40697WIN:cvnLENOVO:ct31:cvrNone:skuLENOVO_MT_20XY_BU_Think_FM_ThinkPadX1YogaGen6:
dmi.product.family: ThinkPad X1 Yoga Gen 6
dmi.product.name: 20XY0027US
dmi.product.sku: LENOVO_MT_20XY_BU_Think_FM_ThinkPad X1 Yoga Gen 6
dmi.product.version: ThinkPad X1 Yoga Gen 6
dmi.sys.vendor: LENOVO
version.compiz: compiz N/A
version.libdrm2: libdrm2 2.4.113-2
version.libgl1-mesa-dri: libgl1-mesa-dri 22.2.5-0ubuntu0.1
version.libgl1-mesa-glx: libgl1-mesa-glx 22.2.5-0ubuntu0.1
version.xserver-xorg-core: xserver-xorg-core 2:21.1.4-2ubuntu1.4
version.xserver-xorg-input-evdev: xserver-xorg-input-evdev N/A
version.xserver-xorg-video-ati: xserver-xorg-video-ati 1:19.1.0-3
version.xserver-xorg-video-intel: xserver-xorg-video-intel 2:2.99.917+git20210115-1
version.xserver-xorg-video-nouveau: xserver-xorg-video-nouveau 1:1.0.17-2build1

[beadon (bryant-eadon)]

process has since been killed -- however I hav statistics posted here :
https://gitlab.gnome.org/GNOME/gnome-software/-/issues/2064

Additional files and memory usage from LSOF report attached.

[beadon (bryant-eadon)]

sorted report.

[beadon (bryant-eadon)]

shoot -- can a maintainer please help me attach this to the right poject ? This is not Xorg specific.

[Daniel van Vugt (vanvugt)]

The attached information looks like memory mapped files. So that's virtual address space, not real memory.

Please run:

  ps auxw > psall.txt

and attach the resulting text file here.

[beadon (bryant-eadon)]

Ok, one better, sorted results (lots of chrome tabs appear, fyi).

ps auxw --sort -rss > psall.txt

See attached file, since gnome-software has restarted, in 1 day it's up to 501+MB and written ~1.9GB to disk (why??).

[Daniel van Vugt (vanvugt)]

There is no indication of a bug in the attachment of comment #6.

The largest process is one of your Chrome tabs, and most of your memory is occupied by Chrome tabs.

gnome-software has a resident set size of "only" 493616K. I agree that's a silly amount of memory, but it's not unusual these days and is not "6+ GB". If gnome-software is growing further over time then please provide more samples showing it getting bigger.

[beadon (bryant-eadon)]

Yes, I killed the process as it approached 6GB after I filed this report.

I will report back with the growth over the next few days. I suspect it occured when I switched between different wireless networks and physical locations, but I would need to run a series of tests to conslude this for certain.

I also find Chrome's dramatic consumption of memory to be unnecessary, but this is another task to trace....

For the moment, I still am focused on gnome-software.

For reference I am finding 2-older reports of a memory leak in gnome-software, but these too seemed to have died out as the reporter eventually stopped responding (they're a year old or more). These might be related.

reference:
https://gitlab.gnome.org/GNOME/gnome-software/-/issues/1590
https://gitlab.gnome.org/GNOME/gnome-software/-/issues/941

[Daniel van Vugt (vanvugt)]

Please also only test the "apt" version of gnome-software for now because bugs from flatpaks can't be tracked here. The "ubuntu/+source" part of the URL means we're only talking about debs installed via apt.

[beadon (bryant-eadon)]

Ok, somewhat good news. I have been watching the memory usage on the machine as many applications start / stop , and am tracking the growth and management.

A tool to tweak the behavior ---
One system parameter I had tweaked before reporting this issue was vm.swappiness , I had moved this down to 30, mostly because I'd like to get the best performance, while avoiding the use of swap space. it had the intened effect, while using almost all available memory - generally a good thing (why have all this memory if you're not using it, right?).

Now, i've moved this back up to 60 (which I believe is the default for the kernel options). In response I am seeing better free-memory management from the kernel. The downside is that I am absolutely seeing lower performance on and across web-pages and other applications as swap is used a bit more. The difference is measured in fractions of a second, I could measure the difference but I'm not sure if this audience would like to see those details (let me know if this is interesting).

For those at home -- edit: /etc/sysctl.conf , modify or add the parameter 'vm.swappiness = 60' and then on your terminal run 'sysctl -p'.

I mention this on this bug thread because this parameter directly effects the OOM killer behavior because the OOM killer appears to be swapiness un-aware, and this is also the underlying behavior modifier for the Chrome threads, and the Flatpak or gnome-software threads as they grow in size. I will continue to monitor for gnome-software threads and memory usage, but this means the heavier memory usage is effectively hidden by worse offenders (chrome likely being one of the heaviest users for us all).

On Chrome --
Since Chrome is doing so much lately to manage their own threads, I hope this kerenel tuning parameter is kept in mind for memory management, it could save the team a lot of time building their own memory management utility/toolchain. There might even be some kind of alert / capability for the kernel to respond differently to certain applications depending on how aggressive they would like their own userspace managed. (Kerenel development feature!)

One gnome-software --
I will keep tabs on the Flatpak memory usage. In the mean time I have disabled snapd entirely. I don't wish to manage my applications like this. Apt is about as far as I want to go now, the interactions of many different pakage managers adds a lot of complexity. I want to incentivize application develoeprs to use as few packaging systems as possible. It's getting quite bad out there - there's are now whole businesses that have sprung up to convert the same binaries into different packaging formats and their dependencies. It's really got to stop, it's such a waste of time and resources to 're-invent the wheel'.

On the root cause --
I would much prefer that we encourage developers to build statically compiled binaries, rather than build containers upon containers to manage what I see as effectively a problem stemming from resource sharing linked to library (mis)management.

I hope this is helpful to development here and across kernel development. I will continue to provide meaningful feedba...