== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 == ---Problem Description--- Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails # virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose root@9.47.70.201's password: Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu' 2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4). Machine Type = Boston ---Steps to Reproduce--- 1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22, power8 2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration 3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected, # virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose Migration: [100 %] 4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken, # virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose root@9.47.70.201's password: Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu' 2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument ---uname output--- Ubuntu 18.04 P9 Host: Kernel # uname -a Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Qemu: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library Ubuntu 16.04.4 P8 Host: Kernel: # uname -a Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Qemu: # dpkg -l | grep qemu ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files) ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc) ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities Libvirt: # dpkg -l | grep libvirt ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings Ubuntu 16.04.4 P8 Guest: Kernel: # uname -a Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux Attachment: 1. Libvirtd debug logs: source and destination host 2. Sosreport: source host, destination host and Guest ---Debugger--- A debugger is not configured Userspace tool common name: qemu Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7 The userspace tool has the following bit modes: ppc64le Userspace tool obtained from project website: na == Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 == Backward migration had three distinct issues. Fixes are now upstream: https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7 https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6 https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well. == Comment: #6 - Balamuruhan S