Mistery memory usage in linux Kernel >= 5.14

Bug #1990292 reported by Federico Foschini
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
linux (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

My hardware is SuperMicro X11SDV-8C-TP8F with Intel D-2146NT.
I've tested this issue with:

* Ubuntu 22.04 Kernel 5.15
* Ubuntu 22.04 Kernel 5.17
* Ubuntu 20.04 Kernel 5.15
* Ubuntu 20.04 Kernel 5.14

Using Ubuntu 20.04 with kernel 5.13 or lower will NOT trigger this issue.

The issue is that I have high RAM usage from the server startup I cannot explain.

At startup without any services active I have nearly 6GiB of RAM used:
```
# free -m
               total used free shared buff/cache available
Mem: 15663 5998 8936 155 728 9213
Swap: 0 0 0
```

Using ps the sum of RSS memory does not adds up: it shows less than 1GiB used:
```
# ps aux | awk '{sum+=$6} END {print sum / 1024}'
944.879
```

This is the meminfo output
```
cat /proc/meminfo
MemTotal: 16039044 kB
MemFree: 8486372 kB
MemAvailable: 9607120 kB
Buffers: 79440 kB
Cached: 1300092 kB
SwapCached: 0 kB
Active: 575856 kB
Inactive: 1141004 kB
Active(anon): 24184 kB
Inactive(anon): 369940 kB
Active(file): 551672 kB
Inactive(file): 771064 kB
Unevictable: 27884 kB
Mlocked: 27884 kB
SwapTotal: 2097148 kB
SwapFree: 2097148 kB
Dirty: 36 kB
Writeback: 0 kB
AnonPages: 365256 kB
Mapped: 304856 kB
Shmem: 50208 kB
KReclaimable: 110308 kB
Slab: 455680 kB
SReclaimable: 110308 kB
SUnreclaim: 345372 kB
KernelStack: 7664 kB
PageTables: 6352 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 10116668 kB
Committed_AS: 2497484 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 31940 kB
VmallocChunk: 0 kB
Percpu: 12800 kB
HardwareCorrupted: 0 kB
AnonHugePages: 4096 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 315700 kB
DirectMap2M: 11919360 kB
DirectMap1G: 6291456 kB
```
And this is the `slabtop` header:
```
 Active / Total Objects (% used) : 1095487 / 1172800 (93.4%)
 Active / Total Slabs (% used) : 28258 / 28258 (100.0%)
 Active / Total Caches (% used) : 121 / 176 (68.8%)
 Active / Total Size (% used) : 401911.16K / 415423.19K (96.7%)
 Minimum / Average / Maximum Object : 0.01K / 0.35K / 10.62K
```

This is the `df -h` output:
```
df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 1.6G 50M 1.5G 4% /run
/dev/mapper/ubuntu--vg-ubuntu--lv 14G 3.8G 9.0G 30% /
tmpfs 7.7G 20K 7.7G 1% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda2 2.0G 127M 1.7G 7% /boot
/dev/mapper/vg0-var_log 9.8G 127M 9.2G 2% /var/log
/dev/mapper/vg0-tmp 9.8G 68K 9.3G 1% /tmp
/dev/mapper/vg0-var_tmp 9.8G 96K 9.3G 1% /var/tmp
/dev/mapper/vg0-var_lib_docker 1.7T 11G 1.6T 1% /var/lib/docker
tmpfs 1.6G 0 1.6G 0% /run/user/2010
tmpfs 1.6G 0 1.6G 0% /run/user/2023
```
---
ProblemType: Bug
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Sep 20 14:54 seq
 crw-rw---- 1 root audio 116, 33 Sep 20 14:54 timer
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu82.1
Architecture: amd64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CRDA: N/A
CasperMD5CheckResult: unknown
DistroRelease: Ubuntu 22.04
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
MachineType: Supermicro SYS-5019D-FN8TP
Package: linux (not installed)
PciMultimedia:

ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcFB: 0 astdrmfb
ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-5.15.0-47-generic root=/dev/mapper/ubuntu--vg-ubuntu--lv ro fsck.repair=yes debian-installer/language=it keyboard-configuration/layoutcode?=it
ProcVersionSignature: Ubuntu 5.15.0-47.51-generic 5.15.46
RelatedPackageVersions:
 linux-restricted-modules-5.15.0-47-generic N/A
 linux-backports-modules-5.15.0-47-generic N/A
 linux-firmware 20220329.git681281e4-0ubuntu3.5
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
Tags: jammy
Uname: Linux 5.15.0-47-generic x86_64
UnreportableReason: This report is about a package that is not installed.
UpgradeStatus: No upgrade log present (probably fresh install)
UserGroups: N/A
_MarkForUpload: False
dmi.bios.date: 05/25/2021
dmi.bios.release: 5.14
dmi.bios.vendor: American Megatrends Inc.
dmi.bios.version: 1.5
dmi.board.asset.tag: To be filled by O.E.M.
dmi.board.name: X11SDV-8C-TP8F
dmi.board.vendor: Supermicro
dmi.board.version: 1.10
dmi.chassis.asset.tag: To be filled by O.E.M.
dmi.chassis.type: 1
dmi.chassis.vendor: Supermicro
dmi.chassis.version: 0123456789
dmi.modalias: dmi:bvnAmericanMegatrendsInc.:bvr1.5:bd05/25/2021:br5.14:svnSupermicro:pnSYS-5019D-FN8TP:pvr0123456789:rvnSupermicro:rnX11SDV-8C-TP8F:rvr1.10:cvnSupermicro:ct1:cvr0123456789:skuTobefilledbyO.E.M.:
dmi.product.family: To be filled by O.E.M.
dmi.product.name: SYS-5019D-FN8TP
dmi.product.sku: To be filled by O.E.M.
dmi.product.version: 0123456789
dmi.sys.vendor: Supermicro

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Libera.chat.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1990292/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

tags: added: bot-comment
Paul White (paulw2u)
affects: ubuntu → linux (Ubuntu)
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1990292

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Revision history for this message
Federico Foschini (undicizeri) wrote : CurrentDmesg.txt

apport information

tags: added: apport-collected jammy
description: updated
Revision history for this message
Federico Foschini (undicizeri) wrote : Lspci.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : Lspci-vt.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : Lsusb.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : Lsusb-t.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : Lsusb-v.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : ProcCpuinfoMinimal.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : ProcInterrupts.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : ProcModules.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : UdevDb.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : WifiSyslog.txt

apport information

Revision history for this message
Federico Foschini (undicizeri) wrote : acpidump.txt

apport information

Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Federico Foschini (undicizeri) wrote :

By removing kernel modules one by one I identified the culprit in "irdma". By removing this module 5.2GiB of RAM is freed

This is the output of "free" without the irdma Kernel module:

free -h
              total used free shared buff/cache available
Mem: 15Gi 289Mi 14Gi 1.0Mi 651Mi 14Gi

Revision history for this message
Federico Foschini (undicizeri) wrote :

Can I provide something else to help the triage of this bug?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.