Writes permanently hang with very heavy I/O on virtio-scsi - worse on virtio-blk
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
QEMU |
Invalid
|
Undecided
|
Unassigned |
Bug Description
Up to date Arch Linux on host and guest. linux 5.2.11. QEMU 4.1.0. Full command line at bottom.
Host gives QEMU two thin LVM volumes. The first is the root filesystem, and the second is for heavy I/O, on a Samsung 970 Evo 1TB.
When maxing out the I/O on the second virtual block device using virtio-blk, I often get a "lockup" in about an hour or two. From the advise of iggy in IRC, I switched over to virtio-scsi. It ran perfectly for a few days, but then "locked up" in the same way.
By "lockup", I mean writes to the second virtual block device permanently hang. I can read files from it, but even "touch foo" never times out, cannot be "kill -9"'ed, and is stuck in uninterruptible sleep.
When this happens, writes to the first virtual block device with the root filesystem are fine, so the O/S itself remains responsive.
The second virtual block device uses BTRFS. But, I have also tried XFS and reproduced the issue.
In guest, when this starts, it starts logging "task X blocked for more than Y seconds". Below is an example of one of these. At this point, anything that is or does in the future write to this block device gets stuck in uninterruptible sleep.
-----
INFO: task kcompactd:232 blocked for more than 860 seconds.
Not tained 5.2.11-1 #1
"echo 0 > /proc/sys/
kcompactd0 D 0 232 2 0x80004000
Call Trace:
? __schedule+
schedule+0x3d/0xc0
io_schedule+
__lock_
? add_to_
migrate_
? isolate_
? __reset_
compact_
kcompactd_
? kvm_clock_
? kvm_sched_
kcompactd+
? wait_woken+
kthread+0xfd/0x130
? kcompactd_
? kthread_
ret_from_
-----
In guest, there are no other dmesg/journalctl entries other than "task...blocked".
On host, there are no dmesg/journalctl entries whatsoever. Everything else in host continues to work fine, including other QEMU VM's on the same underlying SSD (but obviously different lvm volumes.)
I understand there might not be enough to go on here, and I also understand it's possible this isn't a QEMU bug. Happy to run given commands or patches to help diagnose what's going on here.
I'm now running a custom compiled QEMU 4.1.0, with debug symbols, so I can get a meaningful backtrace from the host point of view.
I've only recently tried this level of I/O, so can't say if this is a new issue.
When writes are hanging, on host, I can connect to the monitor. Running "info block" shows nothing unusual.
-----
/usr/bin/
-name arch,process=
-no-user-config
-nodefaults
-nographic
-uuid 0528162b-
-pidfile /tmp/0528162b-
-machine q35,accel=
-cpu SandyBridge-IBRS
-smp cpus=24,
-m 24G
-drive if=pflash,
-drive if=pflash,
-monitor telnet:
-spice unix,addr=
-device ioh3420,
-device virtio-
-usbdevice tablet
-netdev bridge,
-device virtio-
-device virtio-
-drive driver=
-device scsi-hd,
-drive driver=
-device scsi-hd,
-----
summary: |
- irtiWrites permanently hang with very heavy I/O on vo-scsi - worse on + Writes permanently hang with very heavy I/O on virtio-scsi - worse on virtio-blk |
description: | updated |
description: | updated |
On Thu, Sep 05, 2019 at 03:42:03AM -0000, James Harvey wrote: kernel/ hung_task_ timeout_ secs" disables this messae. 0x27f/0x6d0 0x12/0x40 page+0x14a/ 0x250 page_cache_ lru+0xe0/ 0xe0 pages+0x803/ 0xb70 migratepages_ block+0x9f0/ 0x9f0 isolation_ suitable+ 0x110/0x110 zone+0x6a2/ 0xd30 do_work+ 0x134/0x260 read+0x14/ 0x30 clock_read+ 0x5/0x10 0xd3/0x220 0x80/0x80 do_work+ 0x260/0x260 park+0x80/ 0x80 fork+0x35/ 0x40 qemu-system- x86_64 qemu:a. ..
> ** Description changed:
>
> Up to date Arch Linux on host and guest. linux 5.2.11. QEMU 4.1.0.
> Full command line at bottom.
>
> Host gives QEMU two thin LVM volumes. The first is the root filesystem,
> and the second is for heavy I/O, on a Samsung 970 Evo 1TB.
>
> When maxing out the I/O on the second virtual block device using virtio-
> blk, I often get a "lockup" in about an hour or two. From the advise of
> iggy in IRC, I switched over to virtio-scsi. It ran perfectly for a few
> days, but then "locked up" in the same way.
>
> By "lockup", I mean writes to the second virtual block device
> permanently hang. I can read files from it, but even "touch foo" never
> times out, cannot be "kill -9"'ed, and is stuck in uninterruptible
> sleep.
>
> When this happens, writes to the first virtual block device with the
> root filesystem are fine, so the O/S itself remains responsive.
>
> The second virtual block device uses BTRFS. But, I have also tried XFS
> and reproduced the issue.
>
> In guest, when this starts, it starts logging "task X blocked for more
> than Y seconds". Below is an example of one of these. At this point,
> anything that is or does in the future write to this block device gets
> stuck in uninterruptible sleep.
>
> -----
>
> INFO: task kcompactd:232 blocked for more than 860 seconds.
> Not tained 5.2.11-1 #1
> "echo 0 > /proc/sys/
> kcompactd0 D 0 232 2 0x80004000
> Call Trace:
> ? __schedule+
> schedule+0x3d/0xc0
> io_schedule+
> __lock_
> ? add_to_
> migrate_
> ? isolate_
> ? __reset_
> compact_
> kcompactd_
> ? kvm_clock_
> ? kvm_sched_
> kcompactd+
> ? wait_woken+
> kthread+0xfd/0x130
> ? kcompactd_
> ? kthread_
> ret_from_
>
> -----
>
> In guest, there are no other dmesg/journalctl entries other than
> "task...blocked".
>
> On host, there are no dmesg/journalctl entries whatsoever. Everything
> else in host continues to work fine, including other QEMU VM's on the
> same underlying SSD (but obviously different lvm volumes.)
>
> I understand there might not be enough to go on here, and I also
> understand it's possible this isn't a QEMU bug. Happy to run given
> commands or patches to help diagnose what's going on here.
>
> I'm now running a custom compiled QEMU 4.1.0, with debug symbols, so I
> can get a meaningful backtrace from the host point of view.
>
> I've only recently tried this level of I/O, so can't say if this is a
> new issue.
>
> + When writes are hanging, on host, I can connect to the monitor. Running
> + "info block" shows nothing unusual.
> +
> -----
>
> /usr/bin/
> -name arch,process=