shiftfs-lxd.sh test in ubuntu_overlayfs_smoke_test failed because of OOM on 64k AWS a1.medium
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ubuntu-kernel-tests |
New
|
Undecided
|
Unassigned |
Bug Description
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G ram)
The test can either timeout with the 600 second threshold , or fail with instance creation:
* Error: Failed instance creation: websocket: close 1006 (abnormal closure): unexpected EOF
* Error: Failed instance creation: Failed creating instance from image: Unpack failed: Failed to run: unsquashfs -f -d /var/snap/
Or even being killed and therefore leads to an incomplete test report.
OOM can be observed in dmesg, it's probably the reason why we are seeing this kind of unexpected failures:
[13076.574301] systemd invoked oom-killer: gfp_mask=
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+
[13076.574328] show_stack+
[13076.574331] dump_stack_
[13076.574335] dump_stack+
[13076.574337] dump_header+
[13076.574341] oom_kill_
[13076.574346] out_of_
[13076.574350] __alloc_
[13076.574353] __alloc_
[13076.574356] __alloc_
[13076.574358] alloc_pages+
[13076.574363] __page_
[13076.574366] pagecache_
[13076.574369] filemap_
[13076.574371] __do_fault+
[13076.574375] do_read_
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_
[13076.574385] __handle_
[13076.574388] handle_
[13076.574391] do_page_
[13076.574395] do_translation_
[13076.574399] do_mem_
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_
[13076.574409] el0t_64_
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
[13076.574421] Node 0 active_anon:16576kB inactive_
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(169839030
tags: | added: sru-20230904 sru-20231002 |
description: | updated |
description: | updated |