| 2024-05-12 06:08:47 |
Andrea Righi |
description |
autopkgtest-virt-qemu sometimes hangs when running tests on noble images. Originally reported by schopin, who also provided a reproducer:
pull-lp-source -d hello
autopkgtest-buildvm-ubuntu-cloud -r noble
autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
I've been able to reproduce it with debugging enabled:
autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot /path/to/image
It can get stuck during different stages, but AFAICT always during "copydown" operations, log excerpts follow. It may be a coincidence, but this started happening around the time linux-image-6.8.0-11-generic (6.8.0-11.11) migrated to noble. The testbeds I used booted 6.6 but then rebooted into that 6.8 kernel after being upgraded by autopkgtest.
-- logs --
Removing autopkgtest-satdep (0) ...
[...]
autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.g8v75e8g/tests-tree/ /t/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.g8v75e]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout']
autopkgtest-virt-qemu: DBG: +< tar --directory /tmp/autopkgtest.output.g8v75e8g/tests-tree/ --
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.ztmr6f5k/runcmd sh -ec if ! test -d /tmp-
autopkgtest-virt-qemu: DBG: +>?
-- or --
autopkgtest: DBG: sending command to testbed: copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.c9utq3bx/tests-tree/', '--warning=none', '-c', '.', '-f', '-'], ['/tmp/autopkgtest-qemu.qtkcgg5l/runcm]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout', "<_io.BufferedReader name='/dev/null'>", 'devnull_read', <_io.BufferedReader name='/dev/null'>]
autopkgtest-virt-qemu: DBG: +< tar --directory /tmp/autopkgtest.output.c9utq3bx/tests-tree/ --warning=none -c . -f -
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.qtkcgg5l/runcmd sh -ec if ! test -d /tmp/autopkgtest.H8NDfW/build.DLR/src/; then mkdir -- /tmp/autopkgtest.H8NDfW/build.DLR/src/; fi; cd-
autopkgtest-virt-qemu: DBG: +>? |
[Impact]
It seems that kernel 6.8 introduced a regression in the 9pfs related to caching and netfslib, that can cause some user-space apps to read content from files that is not up-to-date (when they are used in a producer/consumer fashion).
It seems that the offending commit is this one:
80105ed2fd27 ("9p: Use netfslib read/write_iter")
Reverting the commit seems to fix the problem. However the actual bug might be in netfslib or how netfslib is used in the 9p context.
The regression has been reported upstream and we are still investigating (https://lore.kernel.org/lkml/Zj0ErxVBE3DYT2Ea@gpd/).
In the meantime it probably makes sense to temporarily revert the commit as a SAUCE patch. Then we will drop the SAUCE patch once we'll have a proper fix upstream.
[Test case]
The following test should complete correctly without any timeout:
pull-lp-source -d hello
autopkgtest-buildvm-ubuntu-cloud -r noble
autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
[Fix]
Revert the following commit (until we have a proper fix upstream):
80105ed2fd27 ("9p: Use netfslib read/write_iter")
[Regression potential]
We may experience other regressions related to 9pfs with this change, however it's quite unlikely to happen since we are reverting a commit, restoring the previous behavior.
[Original bug report]
autopkgtest-virt-qemu sometimes hangs when running tests on noble images. Originally reported by schopin, who also provided a reproducer:
pull-lp-source -d hello
autopkgtest-buildvm-ubuntu-cloud -r noble
autopkgtest -U hello*.dsc -- qemu ./autopkgtest-noble-amd64.img
I've been able to reproduce it with debugging enabled:
autopkgtest -ddd -U hello_2.10-3.dsc -- qemu --debug --show-boot /path/to/image
It can get stuck during different stages, but AFAICT always during "copydown" operations, log excerpts follow. It may be a coincidence, but this started happening around the time linux-image-6.8.0-11-generic (6.8.0-11.11) migrated to noble. The testbeds I used booted 6.6 but then rebooted into that 6.8 kernel after being upgraded by autopkgtest.
-- logs --
Removing autopkgtest-satdep (0) ...
[...]
autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.g8v75e8g/tests-tree/ /t/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.g8v75e]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout']
autopkgtest-virt-qemu: DBG: +< tar --directory /tmp/autopkgtest.output.g8v75e8g/tests-tree/ --
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.ztmr6f5k/runcmd sh -ec if ! test -d /tmp-
autopkgtest-virt-qemu: DBG: +>?
-- or --
autopkgtest: DBG: sending command to testbed: copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: executing copydown /tmp/autopkgtest.output.c9utq3bx/tests-tree/ /tmp/autopkgtest.H8NDfW/build.DLR/src/
autopkgtest-virt-qemu: DBG: ['cmdls', "(['tar', '--directory', '/tmp/autopkgtest.output.c9utq3bx/tests-tree/', '--warning=none', '-c', '.', '-f', '-'], ['/tmp/autopkgtest-qemu.qtkcgg5l/runcm]
autopkgtest-virt-qemu: DBG: ['srcstdin', "<_io.BufferedReader name='/dev/null'>", 'deststdout', "<_io.BufferedReader name='/dev/null'>", 'devnull_read', <_io.BufferedReader name='/dev/null'>]
autopkgtest-virt-qemu: DBG: +< tar --directory /tmp/autopkgtest.output.c9utq3bx/tests-tree/ --warning=none -c . -f -
autopkgtest-virt-qemu: DBG: +> /tmp/autopkgtest-qemu.qtkcgg5l/runcmd sh -ec if ! test -d /tmp/autopkgtest.H8NDfW/build.DLR/src/; then mkdir -- /tmp/autopkgtest.H8NDfW/build.DLR/src/; fi; cd-
autopkgtest-virt-qemu: DBG: +>? |
|
