Block I/O crashes after migration

[Takeshi Sone]

Binary package hint: kvm

1. Start KVM on host A
2. Make heavy I/O in guest, e.g. while true; do dd if=/dev/zero of=testfile bs=1M count=1k; done
3. Live-migrate to host B
4. Guest freezes (almost 100% reproducible)

Guest responds to ping and sysrq, but all processes are locked hard so I can't type a command from console/ssh.

Live migration works well when guest is idle (when skipped step 2).

Both hosts are jaunty, guest is lenny (linux-image-2.6.26-1-686).
With virtio-block, guest hangs forever, but with ide, guest resumes in several ten seconds after resetting ide0 by itself.
Shared storage is LVM logical volume on DRBD.

KVM command line:
kvm -name lenny -vnc none -k ja -daemonize -m 128 -drive file=/dev/starvm/lenny.root,if=virtio,index=0,boot=on -drive file=/dev/starvm/lenny.swap,if=virtio,index=1 -net nic,vlan=0,macaddr=00:1A:4A:7e:23:59,model=virtio -net tap,vlan=0,script=/usr/local/vmm/ifup-br1,downscript=no -serial telnet:10.0.0.2:11430,server,nowait -monitor telnet:10.0.0.2:11431,server,nowait -incoming tcp:10.0.0.2:11432

[Takeshi Sone (ts1)]

Upgraded KVM packages today but same problem.

root@star3:~# dpkg -l |grep kvm
ii kvm 1:84+dfsg-0ubuntu8 Full virtualization on i386 and amd64 hardwa
ii kvm-source 1:84+dfsg-0ubuntu8 Source for the KVM driver

[Takeshi Sone (ts1)]

Same problem on Jaunty guest.
Guest kernel was linux-image-virtual.

[Takeshi Sone (ts1)]

I found a workaround to this problem.
Before starting migration, stop guest (from qemu monitor) and wait 1 second, and cont after migration.
This is not quite "live" migration, but guest kernel happily continues running even under heavy I/O load.
This works for both ide and virtio.

[Dustin Kirkland  (kirkland)]

Thanks a lot for the bug report, and the workaround. I'll try to get a setup where I can test this, and confirm it.

:-Dustin

[Anthony Liguori (anthony-codemonkey)]

http://article.gmane.org/gmane.comp.emulators.kvm.devel/35251

This is a known issue with upstream that's currently being investigated.

[Dustin Kirkland  (kirkland)]

Possibly addressed by this commit, Anthony?

http://git.kernel.org/?p=virt/kvm/qemu-kvm.git;a=commit;h=119f9ed1ad0c3f8c4ab78390bc5d0c55b226dc3d

:-Dustin

[Anthony Liguori (anthony-codemonkey)]

Nope.

[Anthony Liguori (anthony-codemonkey)]

Author: Glauber Costa <email address hidden>
Date: Mon Jul 6 09:32:09 2009 -0400

    flush pending aio requests

    When we finish migration, there may be pending async io requests
    in flight. If we don't flush it before stage3 starting, it might be
    the case that the guest loses it.

    Signed-off-by: Glauber Costa <email address hidden>
    Signed-off-by: Anthony Liguori <email address hidden>

[Takeshi Sone (ts1)]

Thank you!
What are the chances that this fix goes into Karmic?

[Dustin Kirkland  (kirkland)]

This should be fixed in Karmic's qemu-kvm-0.11.

Please reopen if you can reproduce this issue with that package.

:-Dustin

[Michael Tokarev (mjt+launchpad-tls)]

It appears this bug (or similar) is back in 0.12. See Debian bug report #580649 (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=580649 ).