live migration with storage encounter assert(!(bs->open_flags & BDRV_O_INACTIVE)) crashes

[meeho yuen]

The vm guest runs a iotest program, and i migrate it with virsh --copy-storage-all,then the qemu process on the source host happens to crash with the following message:

kvm: block/io.c:1543: bdrv_co_pwritev: Assertion `!(bs->open_flags & 0x0800)' failed.
2017-08-24 11:43:45.919+0000: shutting down, reason=crashed

here is the release:
qemu 2.7 & 2.10.rc3 were tested.
libvirt 3.0.0 & 3.2.0 were tested.

command line:
src_host:virsh migrate --verbose --live --persistent --copy-storage-all vm-core qemu+ssh://dst_host/system

Possible Reason: After bdrv_inactivate_all() was called, mirror_run coroutine still writes the left dirty disk data to remote nbd server, which triggers the assertion. But I don't known how to avoid the problem, help is needed! Thanks.

[Eric Blake (eblake)]

On 08/24/2017 07:59 AM, meeho yuen wrote:
> Public bug reported:
>
> The vm guest runs a iotest program, and i migrate it with virsh --copy-
> storage-all,then the qemu process on the source host happens to crash
> with the following message:
>
> kvm: block/io.c:1543: bdrv_co_pwritev: Assertion `!(bs->open_flags & 0x0800)' failed.
> 2017-08-24 11:43:45.919+0000: shutting down, reason=crashed

> here is the release:
> qemu 2.7 & 2.10.rc3 were tested.

The just-tagged 2.10-rc4 includes a fix that should be addressing that
issue during live migration; can you please re-test with that? (see
also https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04513.html)

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. [phone]
Virtualization: qemu.org | libvirt.org

[meeho yuen (meeho)]

Thank you,I will try it.

[meeho yuen (meeho)]

hi,eblake,the problem still exists on qemu 2.10_rc4,although the possibility is less than before.

kvm: block/io.c:1543: bdrv_co_pwritev: Assertion `!(bs->open_flags & 0x0800)' failed.
2017-08-25 11:08:18.963+0000: shutting down, reason=crashed

[Matthew Schumacher (schu-schu)]

I see the same thing:

2017-12-28 21:36:26.837+0000: initiating migration
qemu-system-x86_64: block/io.c:1537: bdrv_co_pwritev: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed.
2017-12-28 21:36:40.516+0000: shutting down, reason=crashed
~

Running:
QEMU emulator version 2.10.1
libvirtd (libvirt) 3.10.0

[meeho yuen (meeho)]

It seems that the following commit for libvirt fixed the problem.
https://github.com/libvirt/libvirt/blob/960326237764f8970250a3608e7b2b880e030715/src/qemu/qemu_migration.c

[Thomas Huth (th-huth)]

Looking through old bug tickets... is this still an issue with the latest version of QEMU? Or could we close this ticket nowadays?

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]