Issue found on AWS instance a1.medium with 64k flavours on various releases: * L-generic-64k * L-lowlatency-64k * J-generic-64k * J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G ram)
Trying to run the test will cause OOM and unexpected failure: [13076.574301] systemd invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 [13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-88-lowlatency-64k #98-Ubuntu [13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018 [13076.574319] Call trace: [13076.574320] dump_backtrace+0x0/0x204 [13076.574328] show_stack+0x20/0x2c [13076.574331] dump_stack_lvl+0x68/0x84 [13076.574335] dump_stack+0x18/0x34 [13076.574337] dump_header+0x54/0x220 [13076.574341] oom_kill_process+0x22c/0x230 [13076.574346] out_of_memory+0xe4/0x364 [13076.574350] __alloc_pages_may_oom+0x118/0x19c [13076.574353] __alloc_pages_slowpath.constprop.0+0x590/0x840 [13076.574356] __alloc_pages+0x200/0x270 [13076.574358] alloc_pages+0xb4/0x1bc [13076.574363] __page_cache_alloc+0xd4/0xe4 [13076.574366] pagecache_get_page+0x178/0x6c0 [13076.574369] filemap_fault+0x57c/0x944 [13076.574371] __do_fault+0x44/0x170 [13076.574375] do_read_fault+0xe4/0x1e0 [13076.574378] do_fault+0xa8/0x1c0 [13076.574381] handle_pte_fault+0x5c/0x1d0 [13076.574385] __handle_mm_fault+0x104/0x350 [13076.574388] handle_mm_fault+0xf8/0x274 [13076.574391] do_page_fault+0x160/0x560 [13076.574395] do_translation_fault+0x98/0xf0 [13076.574399] do_mem_abort+0x4c/0xc0 [13076.574402] el0_ia+0x98/0x1fc [13076.574406] el0t_64_sync_handler+0x124/0x130 [13076.574409] el0t_64_sync+0x1a4/0x1a8 [13076.574412] Mem-Info: [13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0 active_file:4 inactive_file:19 isolated_file:0 unevictable:454 dirty:0 writeback:0 slab_reclaimable:450 slab_unreclaimable:3151 mapped:131 shmem:261 pagetables:290 bounce:0 kernel_misc_reclaimable:0 free:9608 free_pcp:2 free_cma:8180 [13076.574421] Node 0 active_anon:16576kB inactive_anon:653312kB active_file:256kB inactive_file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:16512kB pagetables:18560kB all_unreclaimable? yes [13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_highatomic:0KB active_anon:1088kB inactive_anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB [13076.574436] lowmem_reserve[]: 0 0 57 57 57 [13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_highatomic:0KB active_anon:15488kB inactive_anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB [13076.574452] lowmem_reserve[]: 0 0 0 0 0 [13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB [13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB [13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=16777216kB [13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=524288kB [13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB [13076.574534] 415 total pagecache pages [13076.574535] 0 pages in swap cache [13076.574537] Swap cache stats: add 0, delete 0, find 0/0 [13076.574539] Free swap = 0kB [13076.574540] Total swap = 0kB [13076.574542] 31232 pages RAM [13076.574543] 0 pages HighMem/MovableOnly [13076.574545] 1812 pages reserved [13076.574546] 8192 pages cma reserved [13076.574547] 0 pages hwpoisoned [13076.574549] Tasks state (memory values in pages): [13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal [13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd [13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd [13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network [13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve [13076.574578] [ 620] 0 620 131 29 393216 0 0 cron [13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon [13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat [13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd [13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen [13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind [13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd [13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd [13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd [13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr [13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd [13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty [13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty [13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd [13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd [13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd [13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam) [13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd [13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash [13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd [13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd [13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash [13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd [13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd [13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed [13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo [13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo [13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg [13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo [13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo [13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh [13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs [13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start [13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd [13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq [13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc [13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs [13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd [13076.574709] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_allowed=0,global_oom,task_memcg=/,task=unsquashfs,pid=6063,uid=0 [13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0 [13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [13082.079115] loop7: detected capacity change from 0 to 8 [13082.314025] audit: type=1400 audit(1698390302.185:63): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd_archive-var-snap-lxd-common-lxd-storage-pools-zfs-images-2291196f955ef93ccfcf536069b999f72a1bf96770ef1752720312584fb34553-rootfs" pid=10037 comm="apparmor_parser"
Issue found on AWS instance a1.medium with 64k flavours on various releases:
* L-generic-64k
* L-lowlatency-64k
* J-generic-64k
* J-lowlatency-64k
It looks like this instance is too small for this test (with just 2G ram)
Trying to run the test will cause OOM and unexpected failure: 0x1100cca( GFP_HIGHUSER_ MOVABLE) , order=0, oom_score_adj=0 88-lowlatency- 64k #98-Ubuntu 0x0/0x204 0x20/0x2c lvl+0x68/ 0x84 0x18/0x34 0x54/0x220 process+ 0x22c/0x230 memory+ 0xe4/0x364 pages_may_ oom+0x118/ 0x19c pages_slowpath. constprop. 0+0x590/ 0x840 pages+0x200/ 0x270 0xb4/0x1bc cache_alloc+ 0xd4/0xe4 get_page+ 0x178/0x6c0 fault+0x57c/ 0x944 0x44/0x170 fault+0xe4/ 0x1e0 pte_fault+ 0x5c/0x1d0 mm_fault+ 0x104/0x350 mm_fault+ 0xf8/0x274 fault+0x160/ 0x560 fault+0x98/ 0xf0 abort+0x4c/ 0xc0 sync_handler+ 0x124/0x130 sync+0x1a4/ 0x1a8
active_ file:4 inactive_file:19 isolated_file:0
unevictable: 454 dirty:0 writeback:0
slab_ reclaimable: 450 slab_unreclaima ble:3151
mapped: 131 shmem:261 pagetables:290 bounce:0
kernel_ misc_reclaimabl e:0
free: 9608 free_pcp:2 free_cma:8180 anon:653312kB active_file:256kB inactive_ file:1216kB unevictable:29056kB isolated(anon):0kB isolated(file):0kB mapped:8384kB dirty:0kB writeback:0kB shmem:16704kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_ stack:16512kB pagetables:18560kB all_unreclaimable? yes highatomic: 0KB active_anon:1088kB inactive_ anon:214208kB active_file:0kB inactive_file:448kB unevictable:0kB writepending:0kB present:1048576kB managed:939776kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:523520kB highatomic: 0KB active_anon:15488kB inactive_ anon:439104kB active_file:256kB inactive_file:896kB unevictable:29056kB writepending:0kB present:950272kB managed:943104kB mlocked:29056kB bounce:0kB free_pcp:128kB local_pcp:128kB free_cma:0kB size=16777216kB size=524288kB size=2048kB constraint= CONSTRAINT_ NONE,nodemask= (null), cpuset= init.scope, mems_allowed= 0,global_ oom,task_ memcg=/ ,task=unsquashf s,pid=6063, uid=0 2.185:63) : apparmor="STATUS" operation= "profile_ remove" profile= "unconfined" name="lxd_ archive- var-snap- lxd-common- lxd-storage- pools-zfs- images- 2291196f955ef93 ccfcf536069b999 f72a1bf96770ef1 752720312584fb3 4553-rootfs" pid=10037 comm="apparmor_ parser"
[13076.574301] systemd invoked oom-killer: gfp_mask=
[13076.574312] CPU: 0 PID: 1 Comm: systemd Tainted: P O 5.15.0-
[13076.574316] Hardware name: Amazon EC2 a1.medium/, BIOS 1.0 11/1/2018
[13076.574319] Call trace:
[13076.574320] dump_backtrace+
[13076.574328] show_stack+
[13076.574331] dump_stack_
[13076.574335] dump_stack+
[13076.574337] dump_header+
[13076.574341] oom_kill_
[13076.574346] out_of_
[13076.574350] __alloc_
[13076.574353] __alloc_
[13076.574356] __alloc_
[13076.574358] alloc_pages+
[13076.574363] __page_
[13076.574366] pagecache_
[13076.574369] filemap_
[13076.574371] __do_fault+
[13076.574375] do_read_
[13076.574378] do_fault+0xa8/0x1c0
[13076.574381] handle_
[13076.574385] __handle_
[13076.574388] handle_
[13076.574391] do_page_
[13076.574395] do_translation_
[13076.574399] do_mem_
[13076.574402] el0_ia+0x98/0x1fc
[13076.574406] el0t_64_
[13076.574409] el0t_64_
[13076.574412] Mem-Info:
[13076.574415] active_anon:259 inactive_anon:10208 isolated_anon:0
[13076.574421] Node 0 active_anon:16576kB inactive_
[13076.574428] Node 0 DMA free:568384kB min:41216kB low:51520kB high:61824kB reserved_
[13076.574436] lowmem_reserve[]: 0 0 57 57 57
[13076.574444] Node 0 Normal free:46528kB min:46848kB low:58560kB high:70272kB reserved_
[13076.574452] lowmem_reserve[]: 0 0 0 0 0
[13076.574460] Node 0 DMA: 6*64kB (UM) 5*128kB (UM) 1*256kB (C) 2*512kB (M) 7*1024kB (UMC) 3*2048kB (UC) 1*4096kB (C) 3*8192kB (UMC) 2*16384kB (UC) 1*32768kB (C) 1*65536kB (C) 1*131072kB (C) 1*262144kB (C) 0*524288kB = 568576kB
[13076.574499] Node 0 Normal: 588*64kB (UM) 53*128kB (UM) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB 0*65536kB 0*131072kB 0*262144kB 0*524288kB = 46464kB
[13076.574526] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574529] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574531] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_
[13076.574534] 415 total pagecache pages
[13076.574535] 0 pages in swap cache
[13076.574537] Swap cache stats: add 0, delete 0, find 0/0
[13076.574539] Free swap = 0kB
[13076.574540] Total swap = 0kB
[13076.574542] 31232 pages RAM
[13076.574543] 0 pages HighMem/MovableOnly
[13076.574545] 1812 pages reserved
[13076.574546] 8192 pages cma reserved
[13076.574547] 0 pages hwpoisoned
[13076.574549] Tasks state (memory values in pages):
[13076.574550] [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[13076.574559] [ 400] 0 400 782 88 393216 0 -250 systemd-journal
[13076.574563] [ 431] 0 431 4549 443 458752 0 -1000 multipathd
[13076.574567] [ 441] 0 441 410 58 327680 0 -1000 systemd-udevd
[13076.574571] [ 579] 100 579 277 62 393216 0 0 systemd-network
[13076.574575] [ 581] 101 581 429 121 458752 0 0 systemd-resolve
[13076.574578] [ 620] 0 620 131 29 393216 0 0 cron
[13076.574582] [ 621] 102 621 159 61 393216 0 -900 dbus-daemon
[13076.574585] [ 629] 0 629 550 211 393216 0 0 networkd-dispat
[13076.574589] [ 632] 104 632 3488 37 393216 0 0 rsyslogd
[13076.574592] [ 633] 0 633 19398 108 655360 0 0 amazon-ssm-agen
[13076.574595] [ 636] 0 636 267 74 327680 0 0 systemd-logind
[13076.574599] [ 708] 114 708 305 43 393216 0 0 chronyd
[13076.574602] [ 711] 114 711 175 43 393216 0 0 chronyd
[13076.574605] [ 821] 0 821 3699 67 458752 0 0 polkitd
[13076.574609] [ 822] 0 822 1756 200 393216 0 0 unattended-upgr
[13076.574612] [ 912] 0 912 258 65 393216 0 -1000 sshd
[13076.574615] [ 1047] 0 1047 107 5 327680 0 0 agetty
[13076.574619] [ 1049] 0 1049 96 13 393216 0 0 agetty
[13076.574622] [ 1675] 0 1675 4709 146 458752 0 0 packagekitd
[13076.574626] [ 2116] 0 2116 320 118 327680 0 0 sshd
[13076.574629] [ 2120] 1000 2120 291 86 393216 0 0 systemd
[13076.574632] [ 2122] 1000 2122 1698 127 393216 0 0 (sd-pam)
[13076.574635] [ 2202] 1000 2202 323 100 327680 0 0 sshd
[13076.574639] [ 2203] 1000 2203 157 64 327680 0 0 bash
[13076.574642] [ 2212] 0 2212 320 118 327680 0 0 sshd
[13076.574645] [ 2257] 1000 2257 323 100 327680 0 0 sshd
[13076.574648] [ 2258] 1000 2258 157 56 393216 0 0 bash
[13076.574651] [ 2648] 0 2648 18710 261 655360 0 -999 containerd
[13076.574655] [ 2936] 0 2936 20298 419 720896 0 -500 dockerd
[13076.574660] [ 3659] 0 3659 2750 75 393216 0 0 zed
[13076.574663] [ 4354] 1000 4354 247 74 393216 0 0 sudo
[13076.574666] [ 4355] 1000 4355 247 50 393216 0 0 sudo
[13076.574670] [ 4356] 0 4356 113 7 327680 0 0 dmesg
[13076.574673] [ 4357] 1000 4357 246 73 458752 0 0 sudo
[13076.574676] [ 4358] 1000 4358 246 49 458752 0 0 sudo
[13076.574679] [ 4359] 0 4359 135 39 393216 0 0 shiftfs-lxd.sh
[13076.574683] [ 4559] 0 4559 2885 12 458752 0 0 lxcfs
[13076.574686] [ 4846] 0 4846 45 8 393216 0 0 daemon.start
[13076.574689] [ 4994] 0 4994 96816 1221 720896 0 0 lxd
[13076.574693] [ 5044] 999 5044 188 27 393216 0 0 dnsmasq
[13076.574698] [ 5932] 0 5932 25230 139 655360 0 0 lxc
[13076.574702] [ 6063] 0 6063 9318 6007 458752 0 0 unsquashfs
[13076.574706] [ 9777] 0 9777 15961 75 589824 0 -900 snapd
[13076.574709] oom-kill:
[13076.574734] Out of memory: Killed process 6063 (unsquashfs) total-vm:596352kB, anon-rss:384448kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:448kB oom_score_adj:0
[13078.957238] oom_reaper: reaped process 6063 (unsquashfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[13082.079115] loop7: detected capacity change from 0 to 8
[13082.314025] audit: type=1400 audit(169839030