On Thu, Sep 05, 2019 at 03:42:03AM -0000, James Harvey wrote:
> ** Description changed:
> 
>   Up to date Arch Linux on host and guest.  linux 5.2.11.  QEMU 4.1.0.
>   Full command line at bottom.
>   
>   Host gives QEMU two thin LVM volumes.  The first is the root filesystem,
>   and the second is for heavy I/O, on a Samsung 970 Evo 1TB.
>   
>   When maxing out the I/O on the second virtual block device using virtio-
>   blk, I often get a "lockup" in about an hour or two.  From the advise of
>   iggy in IRC, I switched over to virtio-scsi.  It ran perfectly for a few
>   days, but then "locked up" in the same way.
>   
>   By "lockup", I mean writes to the second virtual block device
>   permanently hang.  I can read files from it, but even "touch foo" never
>   times out, cannot be "kill -9"'ed, and is stuck in uninterruptible
>   sleep.
>   
>   When this happens, writes to the first virtual block device with the
>   root filesystem are fine, so the O/S itself remains responsive.
>   
>   The second virtual block device uses BTRFS.  But, I have also tried XFS
>   and reproduced the issue.
>   
>   In guest, when this starts, it starts logging "task X blocked for more
>   than Y seconds".  Below is an example of one of these.  At this point,
>   anything that is or does in the future write to this block device gets
>   stuck in uninterruptible sleep.
>   
>   -----
>   
>   INFO: task kcompactd:232 blocked for more than 860 seconds.
>         Not tained 5.2.11-1 #1
>   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this messae.
>   kcompactd0      D    0   232      2 0x80004000
>   Call Trace:
>    ? __schedule+0x27f/0x6d0
>    schedule+0x3d/0xc0
>    io_schedule+0x12/0x40
>    __lock_page+0x14a/0x250
>    ? add_to_page_cache_lru+0xe0/0xe0
>    migrate_pages+0x803/0xb70
>    ? isolate_migratepages_block+0x9f0/0x9f0
>    ? __reset_isolation_suitable+0x110/0x110
>    compact_zone+0x6a2/0xd30
>    kcompactd_do_work+0x134/0x260
>    ? kvm_clock_read+0x14/0x30
>    ? kvm_sched_clock_read+0x5/0x10
>    kcompactd+0xd3/0x220
>    ? wait_woken+0x80/0x80
>    kthread+0xfd/0x130
>    ? kcompactd_do_work+0x260/0x260
>    ? kthread_park+0x80/0x80
>    ret_from_fork+0x35/0x40
>   
>   -----
>   
>   In guest, there are no other dmesg/journalctl entries other than
>   "task...blocked".
>   
>   On host, there are no dmesg/journalctl entries whatsoever.  Everything
>   else in host continues to work fine, including other QEMU VM's on the
>   same underlying SSD (but obviously different lvm volumes.)
>   
>   I understand there might not be enough to go on here, and I also
>   understand it's possible this isn't a QEMU bug.  Happy to run given
>   commands or patches to help diagnose what's going on here.
>   
>   I'm now running a custom compiled QEMU 4.1.0, with debug symbols, so I
>   can get a meaningful backtrace from the host point of view.
>   
>   I've only recently tried this level of I/O, so can't say if this is a
>   new issue.
>   
> + When writes are hanging, on host, I can connect to the monitor.  Running
> + "info block" shows nothing unusual.
> + 
>   -----
>   
>   /usr/bin/qemu-system-x86_64
>      -name arch,process=qemu:arch
>      -no-user-config
>      -nodefaults
>      -nographic
>      -uuid 0528162b-2371-41d5-b8da-233fe61b6458
>      -pidfile /tmp/0528162b-2371-41d5-b8da-233fe61b6458.pid
>      -machine q35,accel=kvm,vmport=off,dump-guest-core=off
>      -cpu SandyBridge-IBRS
>      -smp cpus=24,cores=12,threads=1,sockets=2
>      -m 24G
>      -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/OVMF_CODE.fd
>      -drive if=pflash,format=raw,readonly,file=/var/qemu/0528162b-2371-41d5-b8da-233fe61b6458.fd
>      -monitor telnet:localhost:8000,server,nowait,nodelay
>      -spice unix,addr=/tmp/0528162b-2371-41d5-b8da-233fe61b6458.sock,disable-ticketing
>      -device ioh3420,id=pcie.1,bus=pcie.0,slot=0
>      -device virtio-vga,bus=pcie.1,addr=0
>      -usbdevice tablet
>      -netdev bridge,id=network0,br=br0
>      -device virtio-net-pci,netdev=network0,mac=02:37:de:79:19:09,bus=pcie.0,addr=3
>      -device virtio-scsi-pci,id=scsi1
>      -drive driver=raw,node-name=hd0,file=/dev/lvm/arch_root,if=none,discard=unmap
>      -device scsi-hd,drive=hd0,bootindex=1
>      -drive driver=raw,node-name=hd1,file=/dev/lvm/arch_nvme,if=none,discard=unmap
>      -device scsi-hd,drive=hd1,bootindex=2

Please post backtrace of all QEMU threads when I/O is hung.  You can use
"gdb -p $(pidog qemu-system-x86_64)" to connect GDB and "thread apply
all bt" to produce a backtrace of all threads.

Stefan