Memory Leak in hpet_timer results in unusable machine

[Alexander Bulekov]

Fair warning: this might be specific to QTest (specifically its clock_step) command. This reproducer only works with -accel qtest. Build with --enable-sanitizers to exit once we hit 1G RSS.

export ASAN_OPTIONS=hard_rss_limit_mb=1000
cat << EOF | ./i386-softmmu/qemu-system-i386 -nographic \
-nodefaults -qtest stdio -accel qtest
writeq 0xfed0000e 0x15151515151515f1
clock_step
clock_step
clock_step
clock_step
writeq 0xfed00100 0x5e90c5be00ff5e9e
writeq 0xfed00109 0xffffe0ff5cfec0ff
clock_step
EOF

On my machine it takes around 10 seconds to reach the RSS limit.

Unfortunately, I can't find a way to tell ASAN to log each malloc to figure out whats going on, but running the original fuzzing test case with the libfuzzer -trace_malloc=2 flag, I found that the allocations happen here:

MALLOC[130968] 0x60300069ac90 32
    #0 0x55fa3f615851 in __sanitizer_print_stack_trace (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x2683851)
    #1 0x55fa3f55fe88 in fuzzer::PrintStackTrace() (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x25cde88)
    #2 0x55fa3f5447d6 in fuzzer::MallocHook(void const volatile*, unsigned long) (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x25b27d6)
    #3 0x55fa3f61bbb7 in __sanitizer::RunMallocHooks(void const*, unsigned long) (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x2689bb7)
    #4 0x55fa3f596d75 in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, __asan::AllocType, bool) (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x2604d75)
    #5 0x55fa3f596f7a in __asan::asan_calloc(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*) (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x2604f7a)
    #6 0x55fa3f60d173 in calloc (/home/alxndr/Development/qemu/build/i386-softmmu/qemu-fuzz-i386+0x267b173)
    #7 0x7fb300737548 in g_malloc0 (/usr/lib/x86_64-linux-gnu/libglib-2.0.so.0+0x54548)
    #8 0x55fa40157689 in async_run_on_cpu /home/alxndr/Development/qemu/cpus-common.c:163:10
    #9 0x55fa409fab83 in hpet_timer /home/alxndr/Development/qemu/hw/timer/hpet.c:376:9
    #10 0x55fa416a5751 in timerlist_run_timers /home/alxndr/Development/qemu/util/qemu-timer.c:572:9
    #11 0x55fa3fcfdac4 in qtest_clock_warp /home/alxndr/Development/qemu/softmmu/cpus.c:507:9
    #12 0x55fa3fd65c35 in qtest_process_command /home/alxndr/Development/qemu/softmmu/qtest.c:665:9
    #13 0x55fa3fd5e128 in qtest_process_inbuf /home/alxndr/Development/qemu/softmmu/qtest.c:710:9
    #14 0x55fa3fd5de67 in qtest_server_inproc_recv /home/alxndr/Development/qemu/softmmu/qtest.c:817:9
    #15 0x55fa4142b64b in qtest_sendf /home/alxndr/Development/qemu/tests/qtest/libqtest.c:424:5
    #16 0x55fa4142c482 in qtest_clock_step_next /home/alxndr/Development/qemu/tests/qtest/libqtest.c:864:5
    #17 0x55fa414b12d1 in general_fuzz /home/alxndr/Development/qemu/tests/qtest/fuzz/general_fuzz.c:581:17

It doesn't look like we ever exit out of the loop in timerlist_run_timers, ie timer_list->active_timers is always True.

Info From GDB:
#0 0x0000555558070d31 in address_space_stl_internal (as=0x55555f0e8f20 <address_space_memory>, addr=0x0, val=0x0, attrs=..., result=0x0, endian=DEVICE_LITTLE_ENDIAN) at /home/alxndr/Development/qemu/memory_ldst.inc.c:323
#1 0x0000555558071339 in address_space_stl_le (as=0x55555f0e8f20 <address_space_memory>, addr=0x0, val=0x0, attrs=..., result=0x0) at /home/alxndr/Development/qemu/memory_ldst.inc.c:357
#2 0x000055555a6a6f95 in update_irq (timer=0x61f0000005b8, set=0x1) at /home/alxndr/Development/qemu/hw/timer/hpet.c:210
#3 0x000055555a6ae55f in hpet_timer (opaque=0x61f0000005b8) at /home/alxndr/Development/qemu/hw/timer/hpet.c:386
#4 0x000055555c03d178 in timerlist_run_timers (timer_list=0x60b0000528f0) at /home/alxndr/Development/qemu/util/qemu-timer.c:572
#5 0x000055555c03d6b5 in qemu_clock_run_timers (type=QEMU_CLOCK_VIRTUAL) at /home/alxndr/Development/qemu/util/qemu-timer.c:586
#6 0x0000555558c3d0c4 in qtest_clock_warp (dest=0x3461864) at /home/alxndr/Development/qemu/softmmu/cpus.c:507

-Alex

[Thomas Huth (th-huth)]

Still reproduces with the current git version (commit 7fe7fae8b48e3f9c647)

[Alexander Bulekov (a1xndr)]

I moved this report over to QEMU's new bug tracker on gitlab.com.
Please continue with the discussion here:

https://gitlab.com/qemu-project/qemu/-/issues/538

[Thomas Huth (th-huth)]

Thanks for moving it over! ... let's close this one here on Launchpad now.