virsh -c qemu+ssh://root@source/system migrate --domain client --desturi qemu+ssh://root@destination/system fails

[Thomas Schweikle]

After upgrading from Ubuntu 11.10 to 12.04 migrating a VM from a not migrated host (Ubuntu 11.10) to a migrated host (Ubuntu 12.04) fails:
Only error message found:
qemuMigrationUpdateJobStatus:890 : operation failed: migration job: unexpectedly failed

Ubuntu 11.10 uses: libvirt-bin 0.9.2
Ubuntu 12.04 uses: libvirt-bin 0.9.8

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: libvirt-bin 0.9.8-2ubuntu17
Uname: Linux 3.2.16 x86_64
ApportVersion: 2.0.1-0ubuntu5
Architecture: amd64
Date: Wed Apr 25 14:12:29 2012
InstallationMedia: Ubuntu-Server 10.10 "Maverick Meerkat" - Release amd64 (20101007)
ProcEnviron:
 TERM=screen-bce
 SHELL=/bin/bash
 PATH=(custom, user)
 LANG=de_DE.UTF-8
SourcePackage: libvirt
UpgradeStatus: Upgraded to precise on 2012-04-25 (0 days ago)
mtime.conffile..etc.apparmor.d.usr.lib.libvirt.virt.aa.helper: 2012-04-25T12:40:42.878700
mtime.conffile..etc.default.libvirt.bin: 2011-01-25T16:19:29.769778
mtime.conffile..etc.libvirt.libvirtd.conf: 2012-04-25T13:57:06.138857
mtime.conffile..etc.libvirt.lxc.conf: 2011-01-25T16:11:34.670971
mtime.conffile..etc.libvirt.qemu.conf: 2012-04-25T12:18:38.835193
mtime.conffile..etc.sasl2.libvirt.conf: 2011-01-26T17:57:01

[Serge Hallyn (serge-hallyn)]

Thanks for reporting this bug.

Were you able to migrate VMs between oneiric hosts? That fails for me as well, and I believe both are due to bug 869553 which never got SRUd to oneiric.

[Serge Hallyn (serge-hallyn)]

Never mind, when I remove --tunnelled then I reproduce this bug.

[Serge Hallyn (serge-hallyn)]

/var/log/libvirt/qemu/cdboot.log on the destination host shows:

Unknown savevm section or instance 'hpet' 0
load of migration failed

[Serge Hallyn (serge-hallyn)]

I can reproduce this (at least the "unknown savevm section or instance 'hpet' 0" just doing:

qemu-img create -f qcow2 /mnt/x.qcow2 1G
kvm -hda /mnt/x.qcow2 -vnc :1 -monitor stdio
  savevm /mnt/x.out

on the oneiric host, and then on precise host

kvm -hda /mnt/x.qcow2 -vnc :1 -monitor stdio -S
   loadvm /mnt/x.out

[Serge Hallyn (serge-hallyn)]

Using upstream qemu.git, I get

Unknown ramblock "pc.bios", cannot accept migration
qemu: warning: error while loading state for instance 0x0 of device 'ram'
Error -22 while loading VM state

[Serge Hallyn (serge-hallyn)]

Note this is migrating a vm from qemu-kvm-0.14.1 to qemu-kvm-1.0 and to upstream qemu.git.

[Serge Hallyn (serge-hallyn)]

Going by

http://lists.nongnu.org/archive/html/qemu-devel/2012-03/msg05001.html

it looks like upstream simply does not support live migration between versions 0.15 (and older) and 1.0.

[Serge Hallyn (serge-hallyn)]

Lowering the priority. While I feel it is very unfortunate, shutting down the vm, migrating, and restarting *is* a workaround, and it sounds like the bug itself will become 'wontfix'

[Serge Hallyn (serge-hallyn)]

O1 is an uptodate oneiric system (qemu-kvm 0.14.1)
P1 is an uptodate precise system (qemu-kvm 1.0). Same thing happens with upstream git HEAD from qemu
/mnt is an nfs mount shared between the two

On O1:
root@server-8987:/mnt# kvm -version
QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard
root@server-8987:/mnt# cd /mnt
root@server-8987:/mnt# qemu-img create -f qcow2 x.qcow2 1G
root@server-8987:/mnt# wget http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-businesscard.iso
root@server-8987:/mnt# qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139
(qemu) savevm /mnt/x.out
(qemu) quit

On P1:
root@server-8988:/mnt# cd /mnt
root@server-8988:/mnt# kvm -version
QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
root@server-8988:/mnt# qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139 -S
Could not access KVM kernel module: No such file or directory
failed to initialize KVM: No such file or directory
Back to tcg accelerator.
QEMU 1.0 monitor - type 'help' for more information
(qemu) loadvm /mnt/x.out
Unknown savevm section or instance 'hpet' 0
Error -22 while loading VM state

If I do, on O1,
qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139 -S
(qemu) loadvm /mnt/x.out

that succeeds

[Anthony Liguori (anthony-codemonkey)]

On 04/25/2012 11:29 AM, Serge Hallyn wrote:
> O1 is an uptodate oneiric system (qemu-kvm 0.14.1)
> P1 is an uptodate precise system (qemu-kvm 1.0). Same thing happens with upstream git HEAD from qemu
> /mnt is an nfs mount shared between the two
>
> On O1:
> root@server-8987:/mnt# kvm -version
> QEMU emulator version 0.14.1 (qemu-kvm-0.14.1), Copyright (c) 2003-2008 Fabrice Bellard
> root@server-8987:/mnt# cd /mnt
> root@server-8987:/mnt# qemu-img create -f qcow2 x.qcow2 1G
> root@server-8987:/mnt# wget http://cdimage.debian.org/cdimage/daily-builds/daily/arch-latest/amd64/iso-cd/debian-testing-amd64-businesscard.iso
> root@server-8987:/mnt# qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139
> (qemu) savevm /mnt/x.out
> (qemu) quit
>
> On P1:
> root@server-8988:/mnt# cd /mnt
> root@server-8988:/mnt# kvm -version
> QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
> root@server-8988:/mnt# qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139 -S
> Could not access KVM kernel module: No such file or directory
> failed to initialize KVM: No such file or directory
> Back to tcg accelerator.

This looks like a problem, no?

You can't migrate from tcg to KVM.

> QEMU 1.0 monitor - type 'help' for more information
> (qemu) loadvm /mnt/x.out
> Unknown savevm section or instance 'hpet' 0
> Error -22 while loading VM state

Regards,

Anthony Liguori

>
> If I do, on O1,
> qemu-system-x86_64 -M pc-0.14 -vnc :1 -m 512 -cdrom debian-testing-amd64-businesscard.iso -monitor stdio -hda /mnt/x.qcow2 -net user -net nic,model=rtl8139 -S
> (qemu) loadvm /mnt/x.out
>
> that succeeds
>

[Thomas Schweikle (tps)]

> Were you able to migrate VMs between oneiric hosts?
> That fails for me as well, and I believe both are due to bug 869553 which never got SRUd to oneiric.

Between two oreiric hosts this had worked all the time. It does to between two precise hosts, but not between an oneiric and a precise host. As far as I see, this is related to changes between 0.9.2 and 0.9.8 of libvirtd, rendering this unusable. To make it work again there are two possibilities: migrate oneiric to 0.9.8 (at least) of libvirtd, or downgrade precise to 0.9.2. I'd prefer an upgrade to oneiric *and* precise to use 0.9.11 of libvirtd. Such an upgrade can be done while VM are running.

[Thomas Schweikle (tps)]

Turning off gssapi-auth makes it work again: even between oneiric and precise.

[FL (flepoutre-externe)]

Hi,
How do you turning off gssapi-auth ?

[Aaron Toponce (atoponce-deactivatedaccount)]

He means disabling kerberos authentication in your SSH client config, whether it be globally in /etc/ssh/ssh_config or ~/.ssh/config, assuming you're using qemu+ssh:// to migrate your VMs. Either way, this does not work for me. I can reproduce this error with:

# virsh migrate --live --verbose --persistent --copy-storage-all vm1 qemu+ssh://<email address hidden>/system