ubuntu_blktrace_smoke_test.blktrace-smoke-test fails with "*** buffer overflow detected ***: terminated" on Noble

[Jacob Martin]

Seen on N-nvidia 6.8.0-1008.8 and 6.8.0-1009.9. I was also able to reproduce this on N-generic 6.8.0-36.

Observed on nodes: akis, blanka, gunyolk, hidon, and hinyari. Both arm64 and amd64. The test fails most of the time, but not every time.

02:44:19 DEBUG| [stdout] PASSED (CONFIG_BLK_DEV_IO_TRACE=y in /boot/config-6.8.0-1009-nvidia)
02:44:20 DEBUG| [stdout]
02:44:20 DEBUG| [stdout] Using block device /dev/loop0 for path /home/ubuntu/autotest/client/results/default/ubuntu_blktrace_smoke_test.blktrace-smoke-test/mnt
02:44:20 DEBUG| [stdout]
02:44:20 DEBUG| [stdout] Test regime:
02:44:20 DEBUG| [stdout] dd performing 65536 1K block writes
02:44:20 DEBUG| [stdout] looking for at least 1024 blktrace events
02:44:21 DEBUG| [stdout]
02:44:21 DEBUG| [stdout] Wed Jun 26 02:44:21 UTC 2024: blktrace starting
02:44:21 DEBUG| [stdout] Wed Jun 26 02:44:21 UTC 2024: dd starting
02:44:26 DEBUG| [stdout] Wed Jun 26 02:44:26 UTC 2024: dd stopped
02:44:26 DEBUG| [stdout] Wed Jun 26 02:44:26 UTC 2024: waiting for 10 seconds
02:44:36 DEBUG| [stdout] Wed Jun 26 02:44:36 UTC 2024: blktrace being terminated
02:44:37 DEBUG| [stdout] Wed Jun 26 02:44:37 UTC 2024: blktrace terminated
02:44:37 ERROR| [stderr] *** buffer overflow detected ***: terminated
02:44:38 ERROR| [stderr] /home/ubuntu/autotest/client/tests/ubuntu_blktrace_smoke_test/ubuntu_blktrace_smoke_test.sh: line 169: 8356 Broken pipe cat $TMPFILE
02:44:38 ERROR| [stderr] 8357 Aborted (core dumped) | blkparse -i - > ${TMPFILE}.parsed
02:44:38 DEBUG| [stdout] Wed Jun 26 02:44:38 UTC 2024: blktrace data parsed
02:44:38 DEBUG| [stdout]
02:44:38 DEBUG| [stdout] FAILED (expecting at least 1024 block traces events from the dd process, got 509)
02:44:38 DEBUG| [stdout] FAILED (expecting at least 1024 block read traces events, got 0)
02:44:38 DEBUG| [stdout] FAILED (expecting at least 1024 block write traces events, got 0)
02:44:38 DEBUG| [stdout]
02:44:38 DEBUG| [stdout] Summary: 1 passed, 3 failed

[Jacob Martin (jacobmartin)]

I tried this locally on hinyari and there seems to be an out-of-bounds access issue in the blkparse utility. I'm able to trigger the issue every time on Noble by running blkparse directly with the attached blktrace output file.

In Noble, the memset on line 2026 of blkparse.c triggers a run-time bounds check. Running with Valgrind, I found that the Jammy build also performs out-of-bounds accesses, but they succeed and don't trigger any run-time checks.

Noble Valgrind output:
$ cat blktrace-kernel-trace-10521.tmp | valgrind blkparse -i - > blktrace-kernel-trace-10521.tmp.parsed
...
==11486==
*** buffer overflow detected ***: terminated
==11486==
==11486== Process terminating with default action of signal 6 (SIGABRT)
==11486== at 0x4947628: __pthread_kill_implementation (pthread_kill.c:44)
==11486== by 0x48FCB3B: raise (raise.c:26)
==11486== by 0x48E7DFF: abort (abort.c:79)
==11486== by 0x493ABEF: __libc_message_impl (libc_fatal.c:132)
==11486== by 0x49B81A7: __fortify_fail (fortify_fail.c:24)
==11486== by 0x49B79E3: __chk_fail (chk_fail.c:28)
==11486== by 0x49B8A77: __memset_chk (memset_chk.c:25)
==11486== by 0x10FDEF: UnknownInlinedFun (string_fortified.h:59)
==11486== by 0x10FDEF: UnknownInlinedFun (blkparse.c:2026)
==11486== by 0x10FDEF: UnknownInlinedFun (blkparse.c:2071)
==11486== by 0x10FDEF: show_entries_rb (blkparse.c:2131)
==11486== by 0x10AA1B: UnknownInlinedFun (blkparse.c:2634)
==11486== by 0x10AA1B: UnknownInlinedFun (blkparse.c:2655)
==11486== by 0x10AA1B: main (blkparse.c:2939)
...

Jammy Valgrind output:
$ cat blktrace-kernel-trace-10521.tmp | valgrind blkparse -i - > blktrace-kernel-trace-10521.tmp.parsed
...
==11513== Invalid write of size 8
==11513== at 0x10E8F4: memset (string_fortified.h:71)
==11513== by 0x10E8F4: check_cpu_map (blkparse.c:2026)
==11513== by 0x10E8F4: check_sequence (blkparse.c:2071)
==11513== by 0x10E8F4: show_entries_rb (blkparse.c:2131)
==11513== by 0x10A627: do_pipe (blkparse.c:2634)
==11513== by 0x10A627: do_fifo (blkparse.c:2655)
==11513== by 0x10A627: main (blkparse.c:2939)
==11513== Address 0x4b6dcd0 is 0 bytes after a block of size 0 alloc'd
==11513== at 0x4885250: malloc (vg_replace_malloc.c:442)
==11513== by 0x10E8EB: check_cpu_map (blkparse.c:2025)
==11513== by 0x10E8EB: check_sequence (blkparse.c:2071)
==11513== by 0x10E8EB: show_entries_rb (blkparse.c:2131)
==11513== by 0x10A627: do_pipe (blkparse.c:2634)
==11513== by 0x10A627: do_fifo (blkparse.c:2655)
==11513== by 0x10A627: main (blkparse.c:2939)
...

[Jacob Martin (jacobmartin)]

I reported a bug against the blktrace source package: https://bugs.launchpad.net/ubuntu/+source/blktrace/+bug/2073170.