Ubuntu crashes after running out of memory 16G total

[Jair Bolivar]

Ubuntu 16.04 it's crashing after consuming all 16G of ram.

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.13.0-37-generic 4.13.0-37.42~16.04.1
ProcVersionSignature: Ubuntu 4.13.0-37.42~16.04.1-generic 4.13.13
Uname: Linux 4.13.0-37-generic x86_64
ApportVersion: 2.20.1-0ubuntu2.15
Architecture: amd64
Date: Tue Mar 20 02:52:57 2018
InstallationDate: Installed on 2018-03-12 (7 days ago)
InstallationMedia: Ubuntu-Server 16.04.4 LTS "Xenial Xerus" - Release amd64 (20180228)
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)

[Seth Arnold (seth-arnold)]

The OOM killer went after *getty* and other tasks with roughly one megabyte RSS. Crazy.

Thanks

[Jair Bolivar (jairusan)]

Active / Total Objects (% used) : 667634 / 687626 (97.1%)
 Active / Total Slabs (% used) : 13199 / 13199 (100.0%)
 Active / Total Caches (% used) : 83 / 128 (64.8%)
 Active / Total Size (% used) : 190988.44K / 199387.54K (95.8%)
 Minimum / Average / Maximum Object : 0.01K / 0.29K / 18.62K

  OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
 91098 83812 92% 0.19K 2169 42 17352K dentry
 80852 80443 99% 0.12K 1189 68 9512K kernfs_node_cache
 56000 56000 100% 0.50K 875 64 28000K kmalloc-512
 39420 38463 97% 0.59K 730 54 23360K inode_cache
 36224 36224 100% 0.03K 283 128 1132K kmalloc-32
 33824 33824 100% 0.07K 604 56 2416K Acpi-Operand
 31488 31300 99% 0.06K 492 64 1968K anon_vma_chain
 31168 30217 96% 0.06K 487 64 1948K kmalloc-64
 26832 26832 100% 0.10K 688 39 2752K buffer_head
 26316 23501 89% 0.94K 774 34 24768K xfs_inode
 22528 22528 100% 0.02K 88 256 352K kmalloc-16
 21800 21800 100% 0.20K 545 40 4360K vm_area_struct
 15504 15504 100% 0.08K 304 51 1216K anon_vma
 15198 15198 100% 0.04K 149 102 596K Acpi-Namespace
 14910 14910 100% 0.19K 355 42 2840K kmalloc-192
 14448 14198 98% 0.19K 344 42 2752K cred_jar
 12224 11295 92% 0.25K 191 64 3056K kmalloc-256
 11520 10804 93% 0.66K 240 48 7680K proc_inode_cache
  9728 9728 100% 0.01K 19 512 76K kmalloc-8
 10920 8344 76% 0.57K 195 56 6240K radix_tree_node
  7854 7715 98% 0.09K 187 42 748K kmalloc-96
  6942 6942 100% 0.81K 178 39 5696K fuse_inode
  6118 6028 98% 0.69K 133 46 4256K sock_inode_cache
  5760 5760 100% 0.12K 90 64 720K pid
  6656 5173 77% 0.12K 104 64 832K kmalloc-128
  4896 4802 98% 1.00K 153 32 4896K kmalloc-1024
  4505 4505 100% 0.05K 53 85 212K ftrace_event_field
  3654 3654 100% 0.38K 87 42 1392K mnt_cache
  3542 3542 100% 0.69K 77 46 2464K files_cache
  3542 3207 90% 0.69K 77 46 2464K shmem_inode_cache
  2656 2656 100% 1.00K 83 32 2656K signal_cache
  2208 2208 100% 0.16K 46 48 368K xfs_ili
  2162 2162 100% 0.09K 47 46 188K trace_event_file
  2160 2111 97% 2.00K 135 16 4320K kmalloc-2048
  2601 2057 79% 0.31K 51 51 816K nf_conntrack
  1485 1485 100% 2.06K 99 15 3168K mm_struct
  1472 1472 100% 0.06K 23 64 92K kmem_cache_node
  1470 1470 100% 0.38K 35 42 560K kmem_cache
  1470 1470 100% 2.06K 98 15 3136K sighand_cache
  1360 1360 100% 0.02K 8 170 32K fsnotify_mark_connector
   972 931 95% 7.56K 243 4 7776K task_struct
   680 680 100% 0.94K ...

[Jair Bolivar (jairusan)]

root@ld4sttsvcfw1:~# cat /proc/meminfo
MemTotal: 16065528 kB
MemFree: 694236 kB
MemAvailable: 914736 kB
Buffers: 2264 kB
Cached: 444388 kB
SwapCached: 0 kB
Active: 350736 kB
Inactive: 129120 kB
Active(anon): 36092 kB
Inactive(anon): 8920 kB
Active(file): 314644 kB
Inactive(file): 120200 kB
Unevictable: 3652 kB
Mlocked: 3652 kB
SwapTotal: 0 kB
SwapFree: 0 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 36904 kB
Mapped: 56412 kB
Shmem: 9384 kB
Slab: 201292 kB
SReclaimable: 86684 kB
SUnreclaim: 114608 kB
KernelStack: 7232 kB
PageTables: 2816 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 8032764 kB
Committed_AS: 445852 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 159552 kB
DirectMap2M: 5810176 kB
DirectMap1G: 12582912 kB

[Jair Bolivar (jairusan)]

# dmesg
[ 0.000000] random: get_random_bytes called from start_kernel+0x42/0x50d with crng_init=0
[ 0.000000] Linux version 4.13.0-37-generic (buildd@lcy01-amd64-012) (gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.9)) #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 (Ubuntu 4.13.0-37.42~16.04.1-generic 4.13.13)
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.13.0-37-generic root=UUID=fbd7640e-ebd3-4324-9ee2-6882b0a1fec9 ro
[ 0.000000] KERNEL supported cpus:
[ 0.000000] Intel GenuineIntel
[ 0.000000] AMD AuthenticAMD
[ 0.000000] Centaur CentaurHauls
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[ 0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256
[ 0.000000] x86/fpu: xstate_offset[3]: 832, xstate_sizes[3]: 64
[ 0.000000] x86/fpu: xstate_offset[4]: 896, xstate_sizes[4]: 64
[ 0.000000] x86/fpu: xstate_offset[5]: 960, xstate_sizes[5]: 64
[ 0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]: 512
[ 0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[ 0.000000] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]: 8
[ 0.000000] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[ 0.000000] e820: BIOS-provided physical RAM map:
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009bfff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009c000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000006c5cefff] usable
[ 0.000000] BIOS-e820: [mem 0x000000006c5cf000-0x000000006e7cefff] reserved
[ 0.000000] BIOS-e820: [mem 0x000000006e7cf000-0x000000006f5fefff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000006f5ff000-0x000000006f7fefff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000006f7ff000-0x000000006f7fffff] usable
[ 0.000000] BIOS-e820: [mem 0x000000006f800000-0x000000008fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000047fffffff] usable
[ 0.000000] NX (Execute Disable) protection: active
[ 0.000000] random: fast init done
[ 0.000000] SMBIOS 3.0 present.
[ 0.000000] DMI: Dell Inc. PowerEdge R...

[Jair Bolivar (jairusan)]

root@ld4sttsvcfw1:~# ps auxw
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 37652 5612 ? Ss Mar19 0:04 /sbin/init
root 2 0.0 0.0 0 0 ? S Mar19 0:00 [kthreadd]
root 4 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/0:0H]
root 6 0.0 0.0 0 0 ? S< Mar19 0:00 [mm_percpu_wq]
root 7 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/0]
root 8 0.0 0.0 0 0 ? S Mar19 0:06 [rcu_sched]
root 9 0.0 0.0 0 0 ? S Mar19 0:00 [rcu_bh]
root 10 0.0 0.0 0 0 ? S Mar19 0:00 [migration/0]
root 11 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/0]
root 12 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/0]
root 13 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/1]
root 14 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/1]
root 15 0.0 0.0 0 0 ? S Mar19 0:00 [migration/1]
root 16 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/1]
root 18 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/1:0H]
root 19 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/2]
root 20 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/2]
root 21 0.0 0.0 0 0 ? S Mar19 0:00 [migration/2]
root 22 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/2]
root 24 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/2:0H]
root 25 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/3]
root 26 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/3]
root 27 0.0 0.0 0 0 ? S Mar19 0:00 [migration/3]
root 28 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/3]
root 30 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/3:0H]
root 31 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/4]
root 32 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/4]
root 33 0.0 0.0 0 0 ? S Mar19 0:00 [migration/4]
root 34 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/4]
root 36 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/4:0H]
root 37 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/5]
root 38 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/5]
root 39 0.0 0.0 0 0 ? S Mar19 0:00 [migration/5]
root 40 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/5]
root 42 0.0 0.0 0 0 ? S< Mar19 0:00 [kworker/5:0H]
root 43 0.0 0.0 0 0 ? S Mar19 0:00 [cpuhp/6]
root 44 0.0 0.0 0 0 ? S Mar19 0:00 [watchdog/6]
root 45 0.0 0.0 0 0 ? S Mar19 0:00 [migration/6]
root 46 0.0 0.0 0 0 ? S Mar19 0:00 [ksoftirqd/6]
root 48 0...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1757037

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Launchpad Janitor (janitor)]

[Expired for linux (Ubuntu) because there has been no activity for 60 days.]