| 2016-11-14 08:36:13 |
Mehdi Abaakouk |
bug |
|
|
added bug |
| 2016-11-14 09:13:45 |
Mehdi Abaakouk |
bug task added |
|
cloud-archive |
|
| 2016-11-14 10:24:09 |
Mehdi Abaakouk |
attachment added |
|
2.3-trusty-machine-fix https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1641532/+attachment/4777159/+files/2.3-trusty-machine-fix |
|
| 2016-11-14 12:11:53 |
Christian Ehrhardt |
qemu (Ubuntu): status |
New |
Confirmed |
|
| 2016-11-14 12:11:55 |
Christian Ehrhardt |
qemu (Ubuntu): importance |
Undecided |
Critical |
|
| 2016-11-14 12:12:06 |
Christian Ehrhardt |
bug |
|
|
added subscriber Ubuntu Server Team |
| 2016-11-14 12:13:33 |
Christian Ehrhardt |
summary |
Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration |
machine-types trusty and utopic are not unique (depend on the qemu version) |
|
| 2016-11-14 12:35:57 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
| 2017-01-10 15:46:43 |
Christian Ehrhardt |
bug |
|
|
added subscriber James Page |
| 2017-01-10 15:46:50 |
Christian Ehrhardt |
bug |
|
|
added subscriber Jon Grimm |
| 2017-01-10 15:48:29 |
Christian Ehrhardt |
bug |
|
|
added subscriber cargonza |
| 2017-01-10 15:52:51 |
Christian Ehrhardt |
bug |
|
|
added subscriber Mark Baker |
| 2017-01-17 10:41:02 |
Christian Ehrhardt |
bug task added |
|
qemu-kvm (Ubuntu) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Zesty |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Zesty) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu-kvm (Ubuntu Zesty) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Yakkety |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Yakkety) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu-kvm (Ubuntu Yakkety) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Xenial |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Xenial) |
|
| 2017-01-17 10:41:19 |
Christian Ehrhardt |
bug task added |
|
qemu-kvm (Ubuntu Xenial) |
|
| 2017-01-17 10:41:39 |
Christian Ehrhardt |
bug task deleted |
qemu-kvm (Ubuntu) |
|
|
| 2017-01-17 10:41:53 |
Christian Ehrhardt |
bug task deleted |
qemu-kvm (Ubuntu Xenial) |
|
|
| 2017-01-17 10:41:58 |
Christian Ehrhardt |
bug task deleted |
qemu-kvm (Ubuntu Yakkety) |
|
|
| 2017-01-17 10:42:02 |
Christian Ehrhardt |
bug task deleted |
qemu-kvm (Ubuntu Zesty) |
|
|
| 2017-01-17 10:42:07 |
Christian Ehrhardt |
qemu (Ubuntu Xenial): status |
New |
Confirmed |
|
| 2017-01-17 10:42:10 |
Christian Ehrhardt |
qemu (Ubuntu Yakkety): status |
New |
Confirmed |
|
| 2017-01-17 10:42:12 |
Christian Ehrhardt |
qemu (Ubuntu Zesty): status |
Confirmed |
In Progress |
|
| 2017-01-17 10:42:16 |
Christian Ehrhardt |
qemu (Ubuntu Yakkety): importance |
Undecided |
High |
|
| 2017-01-17 10:42:17 |
Christian Ehrhardt |
qemu (Ubuntu Xenial): importance |
Undecided |
High |
|
| 2017-01-25 15:38:04 |
Corey Bryant |
nominated for series |
|
cloud-archive/liberty |
|
| 2017-01-25 15:38:04 |
Corey Bryant |
bug task added |
|
cloud-archive/liberty |
|
| 2017-01-25 15:38:17 |
Corey Bryant |
cloud-archive: status |
New |
Confirmed |
|
| 2017-01-25 15:38:29 |
Corey Bryant |
cloud-archive/liberty: status |
New |
In Progress |
|
| 2017-01-25 15:38:31 |
Corey Bryant |
cloud-archive/liberty: importance |
Undecided |
Critical |
|
| 2017-01-30 07:18:19 |
Christian Ehrhardt |
qemu (Ubuntu Yakkety): status |
Confirmed |
In Progress |
|
| 2017-01-30 07:18:22 |
Christian Ehrhardt |
qemu (Ubuntu Xenial): status |
Confirmed |
In Progress |
|
| 2017-02-13 10:57:53 |
Launchpad Janitor |
qemu (Ubuntu Zesty): status |
In Progress |
Fix Released |
|
| 2017-02-13 10:57:53 |
Launchpad Janitor |
cve linked |
|
2016-5403 |
|
| 2017-02-13 10:57:53 |
Launchpad Janitor |
cve linked |
|
2016-6351 |
|
| 2017-02-13 10:57:53 |
Launchpad Janitor |
cve linked |
|
2016-6490 |
|
| 2017-02-13 14:19:19 |
Christian Ehrhardt |
attachment added |
|
logs of the full path testing https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1641532/+attachment/4818217/+files/full-path-testing-as-part-of-merge.tgz |
|
| 2017-02-13 14:42:10 |
Christian Ehrhardt |
description |
Hi,
I'm currently live-migrating many VMs from an old server to a new one, and some VM can't be live migrated.
The source host is trusty with qemu-system-x86 1:2.3+dfsg-5ubuntu9.4~cloud2
The destination host is xenial with qemu-system-x86 1:2.5+dfsg-5ubuntu10.6
When the issue occurs, the destination host raises an error [1] and stop the migration process.
The only difference I see between VMs where live migration works and those were it doesn't work is a different machine type.
* migration works when VM have been created with pc-i440fx-vivid
* migration doesn't work when VM have been created with pc-i440fx-utopic
[1] the qemu error report by libvirt on the destination host
2016-11-14 08:25:40.774+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.6), hostname: n7.tetaneutral.net
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm-spice -name a81e7133-9601-4432-86dd-a2401dcad8c2 -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Nehalem -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a81e7133-9601-4432-86dd-a2401dcad8c2 -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=fe2641d2-543a-4d65-b75f-6337bf4b8744,uuid=a81e7133-9601-4432-86dd-a2401dcad8c2' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-a81e7133-9601-4432-86dd-a2401dcad8c2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:disks/a81e7133-9601-4432-86dd-a2401dcad8c2_disk.config:id=openstack-service:key=XXXXXXXXXXXX==:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=none,aio=native' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -drive 'file=rbd:disks/volume-a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3:id=openstack-service:key=XXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3,cache=none,aio=native' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=43,id=hostnet0,vhost=on,vhostfd=45 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:37:fb:ba,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/a81e7133-9601-4432-86dd-a2401dcad8c2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -spice port=5916,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2016-11-14T08:25:44.216364Z qemu-system-x86_64: Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration
2016-11-14T08:25:44.216394Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2016-11-14T08:25:44.216509Z qemu-system-x86_64: load of migration failed: Invalid argument
Cheers, |
[Impact]
* Guests that were created with the Trusty (or Utopic) machine type are
not unique. Due to that on migrations between the qemu versions of
multiple Ubuntu releases migrations fail
* Many migrations work just by luck, one that fails is the Utopic Type on
Cloud-Archive Liberty to Xenial.
* The further one migrates that old guest the more breakage this
accumulates. E.g. a Trusty guest (qemu 2.0) migrated to Xenial
(thinks it is qemu 2.5 type) and from there migrating to Yakkety
(which expects it to be a 2.6 type).
* The fix is minimal and makes the types definition stable across
releases as they were intended
[Test Case]
* Spawn a Guest of Type Utopic (the easiest still supported is Trusty +
Cloud Archive Liberty). And then migrate it to Xenial. Without the
patch migration fail, with the patches it works as both now agree
what the guest definition means. Please do note that both ends of the
migrations have to be fixed to get it working.
* Similar a Guest of type Trusty can with the fixes applied be migrated
X->Y->Z and back to X, while without the fix any backward way (and
probably a future forward way) will fail.
* Note: This is complex and there are many potential combinations - the
testslogs attached have various permutations of those.
It has shown that not only the Utopic type issue that was reported
gets fixed, but several backward migrations as well.
[Regression Potential]
* While it fixes the cases that we know, and as testing showed also
several cases that we didn't know before there are two things we can
not avoid.
1. People have to restart the source guests so that the new fixed
definition will take effect.
But Trusty guests that were already migrated to a Host that has
the error will have to be restarted before they can be migrated
further.
Note: no one has to restart guests on Trusty without Cloud
Archive; there the Trusty type is ok - it is a 2.0 which after
the fix Xenial/Yakkety agree.
2. Restarting the guests after the fix will "downgrade" the virtual
hardware. One can think of the machine types as the HW-revision of
the virtual HW. A Guest that was created as e.g. Trusty these days
on Xenial as incorrectly "too new" virtual HW, restarting the
guest will fix that - but as part of that new attributes that it
incorrectly gained when migrating/moving to the new host will be
taken away (to match the definition the guest had when it was
started)
This is actually a fix, but might appear as a regression to
somebody without knowing what was going on.
Also anybody that "wants" the new HW can just upgrade the machine
type to get it, which is actually recommended anyway [1].
[Other Info]
* This is a complex issue, please catch me (cpaelzer) on IRC if you
need/want to go into detail.
Or for Cloud Archive questions coreycb.
[1]: https://wiki.ubuntu.com/QemuKVMMigration#Upgrade_machine_type
--- original description ---
Hi,
I'm currently live-migrating many VMs from an old server to a new one, and some VM can't be live migrated.
The source host is trusty with qemu-system-x86 1:2.3+dfsg-5ubuntu9.4~cloud2
The destination host is xenial with qemu-system-x86 1:2.5+dfsg-5ubuntu10.6
When the issue occurs, the destination host raises an error [1] and stop the migration process.
The only difference I see between VMs where live migration works and those were it doesn't work is a different machine type.
* migration works when VM have been created with pc-i440fx-vivid
* migration doesn't work when VM have been created with pc-i440fx-utopic
[1] the qemu error report by libvirt on the destination host
2016-11-14 08:25:40.774+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.6), hostname: n7.tetaneutral.net
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm-spice -name a81e7133-9601-4432-86dd-a2401dcad8c2 -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Nehalem -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a81e7133-9601-4432-86dd-a2401dcad8c2 -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=fe2641d2-543a-4d65-b75f-6337bf4b8744,uuid=a81e7133-9601-4432-86dd-a2401dcad8c2' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-a81e7133-9601-4432-86dd-a2401dcad8c2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:disks/a81e7133-9601-4432-86dd-a2401dcad8c2_disk.config:id=openstack-service:key=XXXXXXXXXXXX==:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=none,aio=native' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -drive 'file=rbd:disks/volume-a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3:id=openstack-service:key=XXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3,cache=none,aio=native' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=43,id=hostnet0,vhost=on,vhostfd=45 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:37:fb:ba,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/a81e7133-9601-4432-86dd-a2401dcad8c2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -spice port=5916,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2016-11-14T08:25:44.216364Z qemu-system-x86_64: Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration
2016-11-14T08:25:44.216394Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2016-11-14T08:25:44.216509Z qemu-system-x86_64: load of migration failed: Invalid argument
Cheers, |
|
| 2017-02-13 14:43:21 |
Christian Ehrhardt |
description |
[Impact]
* Guests that were created with the Trusty (or Utopic) machine type are
not unique. Due to that on migrations between the qemu versions of
multiple Ubuntu releases migrations fail
* Many migrations work just by luck, one that fails is the Utopic Type on
Cloud-Archive Liberty to Xenial.
* The further one migrates that old guest the more breakage this
accumulates. E.g. a Trusty guest (qemu 2.0) migrated to Xenial
(thinks it is qemu 2.5 type) and from there migrating to Yakkety
(which expects it to be a 2.6 type).
* The fix is minimal and makes the types definition stable across
releases as they were intended
[Test Case]
* Spawn a Guest of Type Utopic (the easiest still supported is Trusty +
Cloud Archive Liberty). And then migrate it to Xenial. Without the
patch migration fail, with the patches it works as both now agree
what the guest definition means. Please do note that both ends of the
migrations have to be fixed to get it working.
* Similar a Guest of type Trusty can with the fixes applied be migrated
X->Y->Z and back to X, while without the fix any backward way (and
probably a future forward way) will fail.
* Note: This is complex and there are many potential combinations - the
testslogs attached have various permutations of those.
It has shown that not only the Utopic type issue that was reported
gets fixed, but several backward migrations as well.
[Regression Potential]
* While it fixes the cases that we know, and as testing showed also
several cases that we didn't know before there are two things we can
not avoid.
1. People have to restart the source guests so that the new fixed
definition will take effect.
But Trusty guests that were already migrated to a Host that has
the error will have to be restarted before they can be migrated
further.
Note: no one has to restart guests on Trusty without Cloud
Archive; there the Trusty type is ok - it is a 2.0 which after
the fix Xenial/Yakkety agree.
2. Restarting the guests after the fix will "downgrade" the virtual
hardware. One can think of the machine types as the HW-revision of
the virtual HW. A Guest that was created as e.g. Trusty these days
on Xenial as incorrectly "too new" virtual HW, restarting the
guest will fix that - but as part of that new attributes that it
incorrectly gained when migrating/moving to the new host will be
taken away (to match the definition the guest had when it was
started)
This is actually a fix, but might appear as a regression to
somebody without knowing what was going on.
Also anybody that "wants" the new HW can just upgrade the machine
type to get it, which is actually recommended anyway [1].
[Other Info]
* This is a complex issue, please catch me (cpaelzer) on IRC if you
need/want to go into detail.
Or for Cloud Archive questions coreycb.
[1]: https://wiki.ubuntu.com/QemuKVMMigration#Upgrade_machine_type
--- original description ---
Hi,
I'm currently live-migrating many VMs from an old server to a new one, and some VM can't be live migrated.
The source host is trusty with qemu-system-x86 1:2.3+dfsg-5ubuntu9.4~cloud2
The destination host is xenial with qemu-system-x86 1:2.5+dfsg-5ubuntu10.6
When the issue occurs, the destination host raises an error [1] and stop the migration process.
The only difference I see between VMs where live migration works and those were it doesn't work is a different machine type.
* migration works when VM have been created with pc-i440fx-vivid
* migration doesn't work when VM have been created with pc-i440fx-utopic
[1] the qemu error report by libvirt on the destination host
2016-11-14 08:25:40.774+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.6), hostname: n7.tetaneutral.net
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm-spice -name a81e7133-9601-4432-86dd-a2401dcad8c2 -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Nehalem -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a81e7133-9601-4432-86dd-a2401dcad8c2 -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=fe2641d2-543a-4d65-b75f-6337bf4b8744,uuid=a81e7133-9601-4432-86dd-a2401dcad8c2' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-a81e7133-9601-4432-86dd-a2401dcad8c2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:disks/a81e7133-9601-4432-86dd-a2401dcad8c2_disk.config:id=openstack-service:key=XXXXXXXXXXXX==:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=none,aio=native' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -drive 'file=rbd:disks/volume-a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3:id=openstack-service:key=XXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3,cache=none,aio=native' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=43,id=hostnet0,vhost=on,vhostfd=45 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:37:fb:ba,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/a81e7133-9601-4432-86dd-a2401dcad8c2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -spice port=5916,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2016-11-14T08:25:44.216364Z qemu-system-x86_64: Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration
2016-11-14T08:25:44.216394Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2016-11-14T08:25:44.216509Z qemu-system-x86_64: load of migration failed: Invalid argument
Cheers, |
[Impact]
* Guests that were created with the Trusty (or Utopic) machine type are
not unique. Due to that on migrations between the qemu versions of
multiple Ubuntu releases migrations fail
* Many migrations work just by luck, one that fails is the Utopic Type on
Cloud-Archive Liberty to Xenial.
* The further one migrates that old guest the more breakage this
accumulates. E.g. a Trusty guest (qemu 2.0) migrated to Xenial
(thinks it is qemu 2.5 type) and from there migrating to Yakkety
(which expects it to be a 2.6 type).
* The fix is minimal and makes the types definition stable across
releases as they were intended
[Test Case]
* Spawn a Guest of Type Utopic (the easiest still supported is Trusty +
Cloud Archive Liberty). And then migrate it to Xenial. Without the
patch migration fail, with the patches it works as both now agree
what the guest definition means. Please do note that both ends of the
migrations have to be fixed to get it working.
* Similar a Guest of type Trusty can with the fixes applied be migrated
X->Y->Z and back to X, while without the fix any backward way (and
probably a future forward way) will fail.
* Note: This is complex and there are many potential combinations - the
testlogs (comment #15) attached have various permutations of those.
It has shown that not only the Utopic type issue that was reported
gets fixed, but several backward migrations as well.
[Regression Potential]
* While it fixes the cases that we know, and as testing showed also
several cases that we didn't know before there are two things we can
not avoid.
1. People have to restart the source guests so that the new fixed
definition will take effect.
But Trusty guests that were already migrated to a Host that has
the error will have to be restarted before they can be migrated
further.
Note: no one has to restart guests on Trusty without Cloud
Archive; there the Trusty type is ok - it is a 2.0 which after
the fix Xenial/Yakkety agree.
2. Restarting the guests after the fix will "downgrade" the virtual
hardware. One can think of the machine types as the HW-revision of
the virtual HW. A Guest that was created as e.g. Trusty these days
on Xenial as incorrectly "too new" virtual HW, restarting the
guest will fix that - but as part of that new attributes that it
incorrectly gained when migrating/moving to the new host will be
taken away (to match the definition the guest had when it was
started)
This is actually a fix, but might appear as a regression to
somebody without knowing what was going on.
Also anybody that "wants" the new HW can just upgrade the machine
type to get it, which is actually recommended anyway [1].
[Other Info]
* This is a complex issue, please catch me (cpaelzer) on IRC if you
need/want to go into detail.
Or for Cloud Archive questions coreycb.
[1]: https://wiki.ubuntu.com/QemuKVMMigration#Upgrade_machine_type
--- original description ---
Hi,
I'm currently live-migrating many VMs from an old server to a new one, and some VM can't be live migrated.
The source host is trusty with qemu-system-x86 1:2.3+dfsg-5ubuntu9.4~cloud2
The destination host is xenial with qemu-system-x86 1:2.5+dfsg-5ubuntu10.6
When the issue occurs, the destination host raises an error [1] and stop the migration process.
The only difference I see between VMs where live migration works and those were it doesn't work is a different machine type.
* migration works when VM have been created with pc-i440fx-vivid
* migration doesn't work when VM have been created with pc-i440fx-utopic
[1] the qemu error report by libvirt on the destination host
2016-11-14 08:25:40.774+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.6), hostname: n7.tetaneutral.net
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm-spice -name a81e7133-9601-4432-86dd-a2401dcad8c2 -S -machine pc-i440fx-utopic,accel=kvm,usb=off -cpu Nehalem -m 256 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid a81e7133-9601-4432-86dd-a2401dcad8c2 -smbios 'type=1,manufacturer=OpenStack Foundation,product=OpenStack Nova,version=2015.1.2,serial=fe2641d2-543a-4d65-b75f-6337bf4b8744,uuid=a81e7133-9601-4432-86dd-a2401dcad8c2' -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-a81e7133-9601-4432-86dd-a2401dcad8c2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 'file=rbd:disks/a81e7133-9601-4432-86dd-a2401dcad8c2_disk.config:id=openstack-service:key=XXXXXXXXXXXX==:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-ide0-1-1,readonly=on,cache=none,aio=native' -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -drive 'file=rbd:disks/volume-a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3:id=openstack-service:key=XXXXXXXXXXXXXXXXXX:auth_supported=cephx\;none:mon_host=192.168.99.251\:6789\;192.168.99.252\:6789\;192.168.99.253\:6789,format=raw,if=none,id=drive-virtio-disk0,serial=a82bb407-5ccb-4b0e-ba68-a0de1cd58cc3,cache=none,aio=native' -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=43,id=hostnet0,vhost=on,vhostfd=45 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:37:fb:ba,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/a81e7133-9601-4432-86dd-a2401dcad8c2/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -spice port=5916,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
2016-11-14T08:25:44.216364Z qemu-system-x86_64: Unknown ramblock "/rom@etc/acpi/rsdp", cannot accept migration
2016-11-14T08:25:44.216394Z qemu-system-x86_64: error while loading state for instance 0x0 of device 'ram'
2016-11-14T08:25:44.216509Z qemu-system-x86_64: load of migration failed: Invalid argument
Cheers, |
|
| 2017-02-13 21:59:50 |
James Page |
cloud-archive: status |
Confirmed |
Fix Committed |
|
| 2017-02-15 14:35:10 |
Chris J Arges |
qemu (Ubuntu Xenial): status |
In Progress |
Fix Committed |
|
| 2017-02-15 14:35:15 |
Chris J Arges |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-02-15 14:35:23 |
Chris J Arges |
bug |
|
|
added subscriber SRU Verification |
| 2017-02-15 14:35:31 |
Chris J Arges |
tags |
|
verification-needed |
|
| 2017-02-15 14:36:42 |
Chris J Arges |
qemu (Ubuntu Yakkety): status |
In Progress |
Fix Committed |
|
| 2017-02-20 08:03:26 |
Christian Ehrhardt |
tags |
verification-needed |
verification-done verification-done-xenial verification-done-yakkety |
|
| 2017-02-22 20:37:57 |
Launchpad Janitor |
qemu (Ubuntu Xenial): status |
Fix Committed |
Fix Released |
|
| 2017-02-22 20:38:08 |
Brian Murray |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2017-02-22 20:39:22 |
Launchpad Janitor |
qemu (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-02-23 19:46:00 |
James Page |
cloud-archive/liberty: status |
In Progress |
Fix Committed |
|
| 2017-02-23 19:46:01 |
James Page |
tags |
verification-done verification-done-xenial verification-done-yakkety |
verification-done verification-done-xenial verification-done-yakkety verification-liberty-needed |
|
| 2017-03-01 11:40:09 |
Christian Ehrhardt |
tags |
verification-done verification-done-xenial verification-done-yakkety verification-liberty-needed |
verification-done verification-done-xenial verification-done-yakkety verification-liberty-done |
|
| 2017-03-01 12:02:05 |
James Page |
cloud-archive/liberty: status |
Fix Committed |
Fix Released |
|
| 2017-03-01 21:38:05 |
Corey Bryant |
cloud-archive: status |
Fix Committed |
Fix Released |
|
