autopkgtest net is failing on 6.2 kernels with 'System is deadlocked on memory'

[Roxana Nicolescu]

[Impact]
In debian/tests/run-image we instantiate an image with 512Mb of ram, but recent jammy/lunar 6.2 kernels need more than that.

Example of a test failure:
https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/i/initramfs-tools/20230619_135049_ab1cb@/log.gz

'net' test fails with a kernel panic due to out-of-memory.

It also relies on an existing and working cloud image. If the cloud image is not available, the test will silently succeed.

[Fix]

The net autopkgtest was ported to the common test framework in initramfs-tools 0.142ubuntu3. This makes the test more robust by relying on the minimum needed for the integration test.

[Test Plan]

Check that the autopkgtest succeed or test it manually:

1. Download a lunar image:
$ autopkgtest-buildvm-ubuntu-cloud -v -a amd64 -r lunar
2. Run autotestpkg tests from initramfs-tools package. The "net" test is the one that depends on the image instantiated by 'run-image'. The testcase should complete successfully without the kernel panic.
$ autopkgtest initramfs-tools -- qemu autopkgtest-lunar-amd64.img

[Where problems could occur]
The usage of a larger image for the tests could fail if the the test system is low on ram size.

[Roxana Nicolescu (roxanan)]

Lunar fix is attached.

Tested locally after downloading a lunar image:

$ autopkgtest initramfs-tools -- qemu autopkgtest-lunar-amd64.img
It uses the package in updates, therefore this fails:
Begin: Running /scripts/local-bottom ... done.
...
...
[ 17.069410] Call Trace:
[ 17.069504] <TASK>
[ 17.069594] dump_stack_lvl+0x48/0x70
[ 17.069747] dump_stack+0x10/0x20
[ 17.069866] panic+0x37b/0x3c0
[ 17.070029] out_of_memory+0x36c/0x370
[ 17.070226] __alloc_pages_may_oom+0x112/0x1e0
[ 17.070480] __alloc_pages_slowpath.constprop.0+0x4c7/0xa20
[ 17.070761] __alloc_pages+0x31d/0x350
[ 17.070992] alloc_pages+0x90/0x1a0
[ 17.071179] __get_free_pages+0x11/0x50
[ 17.071362] pgd_alloc+0x20/0xc0
[ 17.071548] mm_init+0x18a/0x390
[ 17.071730] dup_mm.constprop.0+0x51/0x120
[ 17.071933] copy_process+0xae3/0x1570
[ 17.072165] kernel_clone+0xbd/0x440
[ 17.072373] __do_sys_clone+0x76/0xb0
[ 17.072587] __x64_sys_clone+0x25/0x40
[ 17.072816] do_syscall_64+0x5b/0x90
[ 17.073027] ? syscall_exit_to_user_mode+0x29/0x50
[ 17.073269] ? do_syscall_64+0x67/0x90
[ 17.073446] ? do_user_addr_fault+0x1e8/0x720
[ 17.073654] ? exit_to_user_mode_prepare+0x30/0xb0
[ 17.073871] ? irqentry_exit_to_user_mode+0x9/0x20
[ 17.074089] ? irqentry_exit+0x43/0x50
[ 17.074322] ? exc_page_fault+0x91/0x1b0
[ 17.074474] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 17.074723] RIP: 0033:0x7fb27d64be07
[ 17.074897] Code: 00 66 90 f3 0f 1e fa 64 48 8b 04 25 10 00 00 00 45 31 c0 31 d2 31 f6 bf 11 00 20 01 4c 8d 90 d0 02 00 00 0
[ 17.075588] RSP: 002b:00007ffc9dba5538 EFLAGS: 00000246 ORIG_RAX: 0000000000000038
[ 17.075976] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb27d64be07
[ 17.076359] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001200011
[ 17.076669] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 17.077004] R10: 00007fb27d567a10 R11: 0000000000000246 R12: 0000000000000001
[ 17.077354] R13: 0000556e56d9f5f0 R14: 0000556e56d9d798 R15: 0000000000000000
[ 17.077722] </TASK>
[ 17.078438] Kernel Offset: 0x2c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 17.079602] ---[ end Kernel panic - not syncing: System is deadlocked on memory ]---
qemu-system-x86_64: terminating on signal 15 from pid 7616 (timeout)
autopkgtest [14:15:54]: test net: -----------------------]
autopkgtest [14:15:55]: test net: - - - - - - - - - - results - - - - - - - - - -
net FAIL non-zero exit status 124
autopkgtest [14:15:55]: @@@@@@@@@@@@@@@@@@@@ summary
amd64-klibc PASS
amd64-busybox PASS
amd64-ata-only PASS
amd64-virtio-only PASS
amd64-separate-usr PASS
amd64-panic-shell PASS
command1 PASS
net FAIL non-zero exit status 124
qemu-system-x86_64: terminating on signal 15 from pid 40557 (/usr/bin/python3)

$ autopkgtest ~/canonical/initramf-tools/initramfs-tools_0.142ubuntu3.dsc -- qemu autopkgtest-lunar-amd64.img
This is the fix and the results are good:

net PASS
autopkgtest [14:32:26]: @@@@@@@@@@@@@@@@@@@@ summary
amd64-klibc PASS
amd64-busyb...

[Roxana Nicolescu (roxanan)]

Jammy fix is attached.

Tested locally after downloading a jammy image and then I installed aws-6.2 kernel:

$ autopkgtest initramfs-tools -- qemu autopkgtest-jammy-amd64.img
It uses the package in updates, therefore this fails:

Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... [ 16.508817] mv invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COM0
[ 16.509481] CPU: 0 PID: 162 Comm: mv Not tainted 6.2.0-24-generic #24-Ubuntu
[ 16.509891] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.0-debian-1.16.0-5 04/01/2014
[ 16.510640] Call Trace:
[ 16.511260] <TASK>
[ 16.511555] dump_stack_lvl+0x48/0x70
[ 16.512040] dump_stack+0x10/0x20
[ 16.512238] dump_header+0x50/0x290
[ 16.512436] oom_kill_process+0x10d/0x1c0
[ 16.512679] out_of_memory+0x101/0x370
[ 16.512904] __alloc_pages_may_oom+0x112/0x1e0
[ 16.513159] __alloc_pages_slowpath.constprop.0+0x4c7/0xa20
[ 16.513545] __alloc_pages+0x31d/0x350
[ 16.513867] __folio_alloc+0x1d/0x60
[ 16.514150] ? policy_node+0x69/0x80
[ 16.514378] vma_alloc_folio+0x9f/0x3d0
[ 16.514749] shmem_alloc_folio+0x7c/0xd0
[ 16.515089] shmem_alloc_and_acct_folio+0x77/0x1c0
[ 16.515362] shmem_get_folio_gfp+0x311/0x7a0
[ 16.515640] ? touch_atime+0x44/0x1c0
[ 16.515947] shmem_write_begin+0x68/0x140
[ 16.516260] generic_perform_write+0xd0/0x220
[ 16.516641] __generic_file_write_iter+0xe8/0x1a0
[ 16.516985] ? apparmor_file_permission+0x85/0x1b0
[ 16.517398] generic_file_write_iter+0x70/0xf0
[ 16.517723] vfs_write+0x251/0x410
[ 16.517998] ksys_write+0x73/0x100
[ 16.518240] __x64_sys_write+0x19/0x30
[ 16.518487] do_syscall_64+0x5b/0x90
[ 16.518780] ? do_syscall_64+0x67/0x90
[ 16.519059] ? exit_to_user_mode_prepare+0x30/0xb0
[ 16.519397] ? syscall_exit_to_user_mode+0x29/0x50
[ 16.519748] ? do_syscall_64+0x67/0x90
[ 16.520049] entry_SYSCALL_64_after_hwframe+0x72/0xdc
[ 16.520575] RIP: 0033:0x7fb923f309e4
[ 16.521240] Code: 15 39 a4 0e 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 80 3d fd 2b 0f 00 00 74 13 8
[ 16.522170] RSP: 002b:00007ffc2527fff8 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 16.522674] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb923f309e4
[ 16.523104] RDX: 0000000000010000 RSI: 00007fb923dd3000 RDI: 000000000000000b
[ 16.523545] RBP: 0000000000010000 R08: 00000000ffffffff R09: 0000000000000000
[ 16.524004] R10: 0000000000000022 R11: 0000000000000202 R12: 000000000000000b
[ 16.524433] R13: 00007fb923dd3000 R14: 0000000000000001 R15: 0000000000010000
[ 16.525104] </TASK>
[ 16.525688] Mem-Info:
[ 16.526175] active_anon:6 inactive_anon:28818 isolated_anon:0
[ 16.526175] active_file:61 inactive_file:40 isolated_file:0
[ 16.526175] unevictable:72423 dirty:3 writeback:0
[ 16.526175] slab_reclaimable:4768 slab_unreclaimable:4347
[ 16.526175] mapped:869 shmem:28384 pagetables:39
[ 16.526175] sec_pagetables:0 bounce:0
[ 16.526175] kernel_misc_reclaimable:0
[ 16.526175] free:1154 free_pcp:0 free_cma:0
[ 16.528299] Node 0 active_anon:24kB inactive_anon:115272kB active_file:244kB inactive_fi...

[Benjamin Drung (bdrung)]

The net autopkgtest is too fragile in my opinion. For example [1] failed to download the cloud-image. So I suggest to better backport the porting of the net autopkgtest to the common test framework from initramfs-tools 0.142ubuntu3.

I can prepare the debdiff if you want.

[1] https://autopkgtest.ubuntu.com/results/autopkgtest-mantic/mantic/amd64/i/initramfs-tools/20230502_101701_0a75f@/log.gz

[Roxana Nicolescu (roxanan)]

Hi Benjamin,
Thanks for the fast feedback.
Yeah, I thought about it when I saw mantic had changed, but I saw it after I created the debdiffs and I was too lazy to change the approach.
I can look into that tomorrow morning, no problem.

[Benjamin Drung (bdrung)]

Thanks. You can find the code here: https://code.launchpad.net/~ubuntu-core-dev/ubuntu/+source/initramfs-tools/+git/initramfs-tools

This is the porting commit: https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/initramfs-tools/commit/?id=acf884d60729460c9afa32e2e41827d1d520b8a4

[Benjamin Drung (bdrung)]

I'll go ahead and prepare a combined debdiff to also address bug #2027636.

[Roxana Nicolescu (roxanan)]

Thanks Benjamin for preparing the debdiff and apologize for not picking this up earlier.

[Steve Langasek (vorlon)]

Hello Roxana, or anyone else affected,

Accepted initramfs-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Roxana, or anyone else affected,

Accepted initramfs-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.140ubuntu13.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Benjamin Drung (bdrung)]

Verified successsful autopkgtest runs:

initramfs-tools/0.140ubuntu13.3 on jammy:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/i/initramfs-tools/20230716_110537_df31f@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/i/initramfs-tools/20230714_224812_f1860@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/armhf/i/initramfs-tools/20230714_215559_96d76@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/ppc64el/i/initramfs-tools/20230714_213915_6880a@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/s390x/i/initramfs-tools/20230715_001131_6f9f5@/log.gz

initramfs-tools/0.142ubuntu2.1 on lunar:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/i/initramfs-tools/20230715_025637_9fc82@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/arm64/i/initramfs-tools/20230714_221530_8c679@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/armhf/i/initramfs-tools/20230714_214804_7cdef@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/i/initramfs-tools/20230714_213745_c92c7@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/s390x/i/initramfs-tools/20230714_231328_b0a19@/log.gz

[Timo Aaltonen (tjaalton)]

Hello Roxana, or anyone else affected,

Accepted initramfs-tools into lunar-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu2.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-lunar to verification-done-lunar. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-lunar. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Roxana, or anyone else affected,

Accepted initramfs-tools into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.140ubuntu13.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

the new upload was for bug 2027636, setting tags back to verified here

[Benjamin Drung (bdrung)]

Verified successsful autopkgtest runs:

initramfs-tools/0.140ubuntu13.4 on jammy:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/amd64/i/initramfs-tools/20230728_115027_a4814@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/arm64/i/initramfs-tools/20230728_114858_98ccd@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/armhf/i/initramfs-tools/20230728_113811_c2c0b@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/ppc64el/i/initramfs-tools/20230728_113831_d4a90@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-jammy/jammy/s390x/i/initramfs-tools/20230728_131700_052e0@/log.gz

initramfs-tools/0.142ubuntu2.2 on lunar:

* amd64: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/amd64/i/initramfs-tools/20230729_085539_78fbb@/log.gz
* arm64: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/arm64/i/initramfs-tools/20230728_120120_3604c@/log.gz
* armhf: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/armhf/i/initramfs-tools/20230728_113422_10e6b@/log.gz
* ppc64el: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/i/initramfs-tools/20230728_113425_06348@/log.gz
* s390x: https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/s390x/i/initramfs-tools/20230728_140252_532d0@/log.gz

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu2.2

---------------
initramfs-tools (0.142ubuntu2.2) lunar; urgency=medium

  * hook-functions: add stusb160x kernel module for Tegra IGX (LP: #2027636)

initramfs-tools (0.142ubuntu2.1) lunar; urgency=medium

  [ Loïc Minier ]
  * Include kernel/drivers/usb/typec/ucsi with MODULES=most along with other
    USB storage drivers; LP: #2027636.

  [ Benjamin Drung ]
  * Port the net autopkgtest to the common test framework. This drops
    depending on downloading a cloud image from the Internet and reduces
    the execution time from 3:19 min down to 0:57 min. (LP: #2027575)

 -- Loïc Minier <email address hidden> Tue, 25 Jul 2023 11:44:57 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.140ubuntu13.4

---------------
initramfs-tools (0.140ubuntu13.4) jammy; urgency=medium

  * hook-functions: add stusb160x kernel module for Tegra IGX (LP: #2027636)

initramfs-tools (0.140ubuntu13.3) jammy; urgency=medium

  [ Loïc Minier ]
  * Include kernel/drivers/usb/typec/ucsi with MODULES=most along with other
    USB storage drivers; LP: #2027636.

  [ Benjamin Drung ]
  * Port the net autopkgtest to the common test framework. This drops
    depending on downloading a cloud image from the Internet and reduces
    the execution time from 3:19 min down to 0:57 min. (LP: #2027575)

 -- Loïc Minier <email address hidden> Tue, 25 Jul 2023 11:54:44 +0200