Migration fails between QEMU 1.5 and QEMU 2.0

[Mathieu Gagné]

==============================================================
SRU Justification:
Note: This fix is not applicable to vivid, because we only support migration from LTS->LTS, so from 12.04 to 14.04. Therefore a fix is only needed for 14.04

Impact: live migration of VMS from a 12.04+UCA host to 14.04 fail.
Regression potential: the patches to fix this are very similar in nature to those already in the archive for bug 1291321. Therefore we are reasonable sure they should be safe.

Test case: See below. (Start a VM of machine type pc-i440fx-1.5 on a 12.04+UCA host, and migrate it to a 14.04 host)
Note, for the fix to work, both qemu and libvirt must have the fix.
==============================================================

This bug looks to be very similar to bug #1291321.

The source host runs 12.04 Precise with Ubuntu Cloud Archive. This means we have access to more recent QEMU packages.

The source host has QEMU 1.5 installed. (the package looks to be backported from 13.10 Saucy by UCA)
The destination host is running 14.04 Trusty with QEMU 2.0.

The machine type used by the instance I tried to live migrate is 'pc-i440fx-1.5'. (which looks to be the new default in QEMU 1.5)

When performing a live migration from QEMU 1.5 to QEMU 2.0, the migration fails with this error on the destination host:

Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 20000 in != 40000
qemu: warning: error while loading state for instance 0x0 of device 'ram'

My hypothesis is that QEMU 1.5 was built against the qemu-kvm source tree. As mentionned in bug #1291321, the machine types defined in qemu-kvm are not compatible with the ones defined in qemu. This means the source 'pc-i440fx-1.5' mahine type is not compatible with the destination 'pc-i440fx-1.5' machine type.

A workaround has been introduced in QEMU 2.0 to allow incoming qemu-kvm instances (pc-1.0) to be properly translated to pc-1.0-precise) using the allow_incoming_qemukvm libvirt config.

But since the default machine type changed between QEMU 1.0 (pc-1.0) and 1.5 (pc-i440fx-1.5), the above workaround doesn't work for people running QEMU 1.5 introduced by UCA.

This means people with QEMU 1.5 on 12.04 Precise are stuck with instances they cannot migrate to a host running the latest LTS without destructive steps.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Serge Hallyn (serge-hallyn)]

I don't think this is something that can be addressed in the Ubuntu archive. It sounds like something that'll need to be fixed in the cloud archive.

[Mathieu Gagné (mgagne)]

Serge, QEMU 2.0 is part of 14.04 Trusty. (which coincidentally happens to be the source of backports in UCA)
Irregardless of UCA, people with 13.10 Saucy won't be able to upgrade to the latest supported LTS.

[Mathieu Gagné (mgagne)]

To make the migration work, you need to:
- Create a qemu-kvm compatible machine type for pc-i440fx-1.5. Lets call it pc-i440fx-1.5-saucy.
- Override cirrus-vga.vgamem_mb to 8mb (instead of 10mb)
- Override virtio-net-pci.romfile to use pxe-virtio.rom (instead of efi-virtio.rom)
- Patch libvirt to translate pc-i440fx-1.5 to pc-i440fx-1.5-saucy if allow_incoming_qemukvm is enabled. (just like with pc-1.0)

[Mathieu Gagné (mgagne)]

Patches I made and tested: https://gist.github.com/mgagne/95046681c59e4e20989c

[Serge Hallyn (serge-hallyn)]

@mgagne,

Just an update, I will push your patches as soon as bug 1262641 is verified. Until that happens a new libvirt version cannot be uploaded to trusty-proposed. Hopefully we can get it pushed next week.

[gustavo panizzo (gfa)]

@serge-hallyn, #1262641 has been verified and released, will you push this one?

[Serge Hallyn (serge-hallyn)]

Yes, I will, hopefully tomorrow.

[Chris J Arges (arges)]

Hello Mathieu, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.11 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Mathieu, or anyone else affected,

Accepted qemu into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/2.0.0+dfsg-2ubuntu1.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Chris J Arges (arges)]

Uploaded qemu_2.0.0+dfsg-2ubuntu1.14 since the last patch got dropped. Once this is accepted please verify.

[Chris J Arges (arges)]

Hello Mathieu, or anyone else affected,

Accepted qemu into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/2.0.0+dfsg-2ubuntu1.14 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (xtrusia)]

I had same problem with -proposed pkg

please refer below link

http://pastebin.ubuntu.com/11736355/

[Serge Hallyn (serge-hallyn)]

In your paste you only show qemu versions. To be sure, were you also using the proposed libvirt version (https://launchpad.net/ubuntu/+source/qemu/2.0.0+dfsg-2ubuntu1.12 ) ?

[Chris J Arges (arges)]

Hello Mathieu, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.12 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (xtrusia)]

Yeah I miss that.

I tested this again with libvirt-bin, qemu

Please refere below link

http://pastebin.ubuntu.com/11754311/

[Serge Hallyn (serge-hallyn)]

Sorry, there was some confusion here. The proposed fix for this bug was removed since it failed. Unfortunately the bug# had to stay in the changelog so this bug was updated suggesting there was a new fix for it. Rather libvirt 1.2.2-0ubuntu13.1.12 drops the proposed patchset for this bug, and the proposed qemu version will be deleted or simply superseded without this fix.

(So comment #15 was a mistake)

[Mathieu Gagné (mgagne)]

Seyeong Kim (xtrusia):

You need to test with the following setup:
* The source needs to run qemu 1.5. (which should be unpatched)
* The destination needs to run a *PATCHED* versions of qemu 2.0 and libvirt 1.2. Furthermore, libvirt *NEEDS* to have allow_incoming_qemukvm set to true.

Failure to set allow_incoming_qemukvm to true will result in this error:

Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 20000 in != 40000
qemu: warning: error while loading state for instance 0x0 of device 'ram'

Your logs show that you failed to follow the test steps as the destination isn't using the "pc-i440fx-1.5-saucy" machine type as expected.

Make sure libvirt is properly configured and restarted with "allow_incoming_qemukvm".

The proposed patches (both libvirt and qemu) have been running in production environment without any issue since the bug report.

[Seyeong Kim (xtrusia)]

Mathieu Gagné (mgagne)

I think libvirt 1.2.2-0ubuntu13.1.12(in proposed repository) has problem as #17.

I checked code of libvirt 1.2.2-0ubuntu13.1.12 but there is no fix you made https://gist.github.com/mgagne/95046681c59e4e20989c

[Serge Hallyn (serge-hallyn)]

Quoting Seyeong Kim (<email address hidden>):
> Mathieu Gagné (mgagne)
>
> I think libvirt 1.2.2-0ubuntu13.1.12(in proposed repository) has problem
> as #17.

Yes, 1.2.2-0ubuntu13.1.11 had the fix. You reported that as not working, so
we reverted the fix in 1.2.2-0ubuntu13.1.12. So it does NOT have the fix.

We will have to wait until 1.2.2-0ubuntu13.1.13 for another attempt at this
fix.

[Seyeong Kim (xtrusia)]

ah.. sorry for making confusion

[gustavo panizzo (gfa)]

qemu 2.0.0+dfsg-2ubuntu1.13 does NOT have the patch to live migrate from 1.5.
qemu 2.0.0+dfsg-2ubuntu1.14 in -proposed has the patch

i just moved vm to an hypervisor using 2.0.0+dfsg-2ubuntu1.14 and it works fine, 2.0.0+dfsg-2ubuntu1.12 worked fine in my environment before.

i will comment if .14 generates any issue in my env

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.2-0ubuntu13.1.12

---------------
libvirt (1.2.2-0ubuntu13.1.12) trusty-proposed; urgency=medium

  * Drop Support-incoming-migration-from-13.10-hosts.patch as it failed
    verification.

libvirt (1.2.2-0ubuntu13.1.11) trusty-proposed; urgency=medium

  * Support-incoming-migration-from-13.10-hosts.patch (LP: #1425619)
  * qemu-filterref-crash.patch: fix crash when removing filterref from
    interfaces (LP: #1448205)
  * storage_backend_rbd-correct-arg-order-to-rbd_create3: fix reversed
    arguments to rbd_create3. (LP: #1447030)

 -- Serge Hallyn <email address hidden> Thu, 18 Jun 2015 14:21:06 -0500

[Chris J Arges (arges)]

The verification of the Stable Release Update for libvirt has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Chris J Arges (arges)]

Note, this wasn't really released into trusty yet. I'm going to include it in .13...

[Chris J Arges (arges)]

Just uploaded libvirt_1.2.2-0ubuntu13.1.13 which has this patch re-enabled. Once this is accepted into -proposed let's verify this again. : )

[Chris J Arges (arges)]

Hello Mathieu, or anyone else affected,

Accepted libvirt into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/1.2.2-0ubuntu13.1.13 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Seyeong Kim (xtrusia)]

root@trusty:~# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20131111.c3d1e78-2ubuntu1.1 all PXE boot firmware - ROM images for qemu
ii qemu 2.0.0+dfsg-2ubuntu1.14 amd64 fast processor emulator
ii qemu-keymaps 2.0.0+dfsg-2ubuntu1.14 all QEMU keyboard maps
ii qemu-kvm 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU Full virtualization on x86 hardware (transitional package)
ii qemu-system 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries
ii qemu-system-arm 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU full system emulation binaries (x86)
ii qemu-user 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU user mode emulation binaries
ii qemu-utils 2.0.0+dfsg-2ubuntu1.14 amd64 QEMU utilities
root@trusty:~# dpkg -l | grep libvirt
ii libvirt-bin 1.2.2-0ubuntu13.1.13 amd64 programs for the libvirt library
ii libvirt0 1.2.2-0ubuntu13.1.13 amd64 library for interfacing with different virtualization systems

2015-07-08 14:54:19.979+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name test -S -machine pc-i440fx-1.5-saucy,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid b88fd65b-87f6-6df7-e570-9b4ca6139f23 -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/test.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/xtrusia/test.img,if=none,id=drive-ide0-0-0,format=raw -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=24,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:86:94:15,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -incoming tcp:[::]:49152 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4
char device r...

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 2.0.0+dfsg-2ubuntu1.14

---------------
qemu (2.0.0+dfsg-2ubuntu1.14) trusty; urgency=medium

  * ubuntu/Add-machine-type-pc-i440fx-1.5-qemu-kvm-for-live-migrate.patch:
    enable migration from 13.10 hosts (LP: #1425619)

 -- Chris J Arges <email address hidden> Mon, 15 Jun 2015 12:26:17 -0500

[Launchpad Janitor (janitor)]

This bug was fixed in the package libvirt - 1.2.2-0ubuntu13.1.14

---------------
libvirt (1.2.2-0ubuntu13.1.14) trusty; urgency=medium

  [ Seyeong Kim ]
  * d/p/fix_libvirtd_killed_by_sigsegv.patch: fix incorrect backport
    (LP: #1464175)

 -- Chris J Arges <email address hidden> Wed, 08 Jul 2015 10:52:41 -0500

[James Page (james-page)]

Bug catchup - Marking fix released as all cloud-archive pockets are up-to-date with libvirt and qemu fixes.