== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
power8
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU.