| 2018-12-19 07:49:15 |
bugproxy |
bug |
|
|
added bug |
| 2018-12-19 07:49:17 |
bugproxy |
tags |
|
architecture-ppc64le bugnameltc-167401 severity-high targetmilestone-inin--- |
|
| 2018-12-19 07:49:18 |
bugproxy |
ubuntu: assignee |
|
Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) |
|
| 2018-12-19 07:49:22 |
bugproxy |
affects |
ubuntu |
qemu (Ubuntu) |
|
| 2018-12-19 07:51:49 |
Frank Heimes |
bug task added |
|
ubuntu-power-systems |
|
| 2018-12-19 07:52:16 |
Frank Heimes |
ubuntu-power-systems: importance |
Undecided |
High |
|
| 2018-12-19 07:52:27 |
Frank Heimes |
ubuntu-power-systems: assignee |
|
Canonical Server Team (canonical-server) |
|
| 2019-01-07 15:20:41 |
Christian Ehrhardt |
tags |
architecture-ppc64le bugnameltc-167401 severity-high targetmilestone-inin--- |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- |
|
| 2019-01-07 15:21:40 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-01-07 15:21:40 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Bionic) |
|
| 2019-01-07 15:21:40 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-01-07 15:21:40 |
Christian Ehrhardt |
bug task added |
|
qemu (Ubuntu Cosmic) |
|
| 2019-01-07 15:21:45 |
Christian Ehrhardt |
qemu (Ubuntu): status |
New |
In Progress |
|
| 2019-01-07 15:21:48 |
Christian Ehrhardt |
qemu (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-01-07 15:21:50 |
Christian Ehrhardt |
qemu (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-01-07 15:22:10 |
Andrew Cloke |
ubuntu-power-systems: status |
New |
Triaged |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
qemu (Ubuntu): status |
In Progress |
Fix Released |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2016-10028 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2017-8284 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2017-9060 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-10839 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-11806 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-12617 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-15746 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-16847 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-16867 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-16872 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-17958 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-17962 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-17963 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-18438 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-18849 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-18954 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-19364 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-19489 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-19665 |
|
| 2019-01-30 22:24:10 |
Launchpad Janitor |
cve linked |
|
2018-3639 |
|
| 2019-01-31 07:06:34 |
Frank Heimes |
ubuntu-power-systems: status |
Triaged |
In Progress |
|
| 2019-01-31 08:42:15 |
Christian Ehrhardt |
description |
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
[Impact]
* Users migrating between 16.04 on power8 and 18.04 on poer9 can not
migrate back (the very same guest)
* IBM identified a set of qemu fixes that will make the migration work
as it is not impossible but mostly due to a misadvertisement in
features in the p8 compat mode.
[Test Case]
* Get two hosts:
- Ubuntu 16.04 on power8
- Ubuntu 18.04 on power9
* on 16.04 spawn a guest and set power 8 compat in libvirt
<cpu mode='host-model'>
<model>power8</model>
</cpu>
* Migrate the guest to the 18.04 host
* Migrate the guest back to the 16.04 host
[Regression Potential]
* The changes are ppc64 only and identified/suggested by IBM so any
fallout should be limited to ppc64 which is their domain
* Furthermore it only affects pre 2.8 guest types, so anything spawned
on 18.04 is unaffected by the changes (as it will default to 2.11)
* The remaining regression "potential" would be around guests migrated
from 16.04 to 18.04 that after the update behave slightly different
(more correct, but different)
[Other Info]
* Fixes are only needed on the newer qemu to handle older guests (pre
2.8 and xenial is 2.5) with power8 compat in a way that they can
be migrated back.
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
|
| 2019-01-31 15:53:57 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/362539 |
|
| 2019-01-31 15:56:49 |
Launchpad Janitor |
merge proposal linked |
|
https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/362540 |
|
| 2019-02-01 12:52:35 |
Christian Ehrhardt |
description |
[Impact]
* Users migrating between 16.04 on power8 and 18.04 on poer9 can not
migrate back (the very same guest)
* IBM identified a set of qemu fixes that will make the migration work
as it is not impossible but mostly due to a misadvertisement in
features in the p8 compat mode.
[Test Case]
* Get two hosts:
- Ubuntu 16.04 on power8
- Ubuntu 18.04 on power9
* on 16.04 spawn a guest and set power 8 compat in libvirt
<cpu mode='host-model'>
<model>power8</model>
</cpu>
* Migrate the guest to the 18.04 host
* Migrate the guest back to the 16.04 host
[Regression Potential]
* The changes are ppc64 only and identified/suggested by IBM so any
fallout should be limited to ppc64 which is their domain
* Furthermore it only affects pre 2.8 guest types, so anything spawned
on 18.04 is unaffected by the changes (as it will default to 2.11)
* The remaining regression "potential" would be around guests migrated
from 16.04 to 18.04 that after the update behave slightly different
(more correct, but different)
[Other Info]
* Fixes are only needed on the newer qemu to handle older guests (pre
2.8 and xenial is 2.5) with power8 compat in a way that they can
be migrated back.
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
[Impact]
* Users migrating between 16.04 on power8 and 18.04 on poer9 can not
migrate back (the very same guest)
* IBM identified a set of qemu fixes that will make the migration work
as it is not impossible but mostly due to a misadvertisement in
features in the p8 compat mode.
[Test Case]
* Get two hosts:
- Ubuntu 16.04 on power8
- remember that on p8 you need to disable smt
$ ppc64_cpu --smt=off
- Ubuntu 18.04 on power9
* create the same guest on both
$ uvt-kvm create --password=ubuntu b-x release=bionic arch=ppc64el label=daily
Shut it down and undefine it on the 18.04 box (that is only to lay
out the files)
$ virsh shutdown b-x
$ virsh undefine b-x
* Shut down the guest on 16.04 and mark it as explcit p8 compat
in libvirt
<cpu mode='host-model'>
<model fallback='allow'>power8</model>
</cpu>
* Backwards migration is a best-effort but not fully supported feature,
so --copy-storage-all won't work as there is a nbd issue between qemu
2.5/2.11, so you'll need shared storage
-> set up shared storage and modify the guest to use that
-> for the sake of simplifying testing you can use copy-storage-all
from X->B but not reverse (and ignore potential minor I/Os being
lost)
* Migrate the guest to the 18.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.162/system
* Migrate the guest back to the 16.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.181/system
[Regression Potential]
* The changes are ppc64 only and identified/suggested by IBM so any
fallout should be limited to ppc64 which is their domain
* Furthermore it only affects pre 2.8 guest types, so anything spawned
on 18.04 is unaffected by the changes (as it will default to 2.11)
* The remaining regression "potential" would be around guests migrated
from 16.04 to 18.04 that after the update behave slightly different
(more correct, but different)
[Other Info]
* Fixes are only needed on the newer qemu to handle older guests (pre
2.8 and xenial is 2.5) with power8 compat in a way that they can
be migrated back.
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
|
| 2019-02-01 12:59:10 |
Christian Ehrhardt |
description |
[Impact]
* Users migrating between 16.04 on power8 and 18.04 on poer9 can not
migrate back (the very same guest)
* IBM identified a set of qemu fixes that will make the migration work
as it is not impossible but mostly due to a misadvertisement in
features in the p8 compat mode.
[Test Case]
* Get two hosts:
- Ubuntu 16.04 on power8
- remember that on p8 you need to disable smt
$ ppc64_cpu --smt=off
- Ubuntu 18.04 on power9
* create the same guest on both
$ uvt-kvm create --password=ubuntu b-x release=bionic arch=ppc64el label=daily
Shut it down and undefine it on the 18.04 box (that is only to lay
out the files)
$ virsh shutdown b-x
$ virsh undefine b-x
* Shut down the guest on 16.04 and mark it as explcit p8 compat
in libvirt
<cpu mode='host-model'>
<model fallback='allow'>power8</model>
</cpu>
* Backwards migration is a best-effort but not fully supported feature,
so --copy-storage-all won't work as there is a nbd issue between qemu
2.5/2.11, so you'll need shared storage
-> set up shared storage and modify the guest to use that
-> for the sake of simplifying testing you can use copy-storage-all
from X->B but not reverse (and ignore potential minor I/Os being
lost)
* Migrate the guest to the 18.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.162/system
* Migrate the guest back to the 16.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.181/system
[Regression Potential]
* The changes are ppc64 only and identified/suggested by IBM so any
fallout should be limited to ppc64 which is their domain
* Furthermore it only affects pre 2.8 guest types, so anything spawned
on 18.04 is unaffected by the changes (as it will default to 2.11)
* The remaining regression "potential" would be around guests migrated
from 16.04 to 18.04 that after the update behave slightly different
(more correct, but different)
[Other Info]
* Fixes are only needed on the newer qemu to handle older guests (pre
2.8 and xenial is 2.5) with power8 compat in a way that they can
be migrated back.
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
[Impact]
* Users migrating between 16.04 on power8 and 18.04 on poer9 can not
migrate back (the very same guest)
* IBM identified a set of qemu fixes that will make the migration work
as it is not impossible but mostly due to a misadvertisement in
features in the p8 compat mode.
[Test Case]
* Get two hosts:
- Ubuntu 16.04 on power8
- remember that on p8 you need to disable smt
$ ppc64_cpu --smt=off
- Ubuntu 18.04 on power9
* create the same guest on both
$ uvt-kvm create --password=ubuntu b-x release=bionic arch=ppc64el label=daily
Shut it down and undefine it on the 18.04 box (that is only to lay
out the files)
$ virsh shutdown b-x
$ virsh undefine b-x
* Shut down the guest on 16.04 and mark it as explcit p8 compat
in libvirt
<cpu mode='host-model'>
<model fallback='allow'>power8</model>
</cpu>
* Backwards migration is a best-effort but not fully supported feature,
so --copy-storage-all won't work as there is a nbd issue between qemu
2.5/2.11, so you'll need shared storage
-> set up shared storage and modify the guest to use that
-> for the sake of simplifying testing you can use copy-storage-all
from X->B but not reverse (and ignore potential minor I/Os being
lost)
* Migrate the guest to the 18.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.162/system
* Migrate the guest back to the 16.04 host
$ virsh migrate --live --verbose b-x qemu+ssh://10.245.71.181/system
Without the fix on the way back you will encounter:
Migration: [100 %]error: operation failed: job: unexpectedly failed
And in the log:
qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
qemu-system-ppc64: load of migration failed: Invalid argument
With the fix the migration works as expected
[Regression Potential]
* The changes are ppc64 only and identified/suggested by IBM so any
fallout should be limited to ppc64 which is their domain
* Furthermore it only affects pre 2.8 guest types, so anything spawned
on 18.04 is unaffected by the changes (as it will default to 2.11)
* The remaining regression "potential" would be around guests migrated
from 16.04 to 18.04 that after the update behave slightly different
(more correct, but different)
[Other Info]
* Fixes are only needed on the newer qemu to handle older guests (pre
2.8 and xenial is 2.5) with power8 compat in a way that they can
be migrated back.
== Comment: #0 - Balamuruhan S - 2018-05-02 08:09:21 ==
---Problem Description---
Backward migration of Ubuntu 16.04.4 P8 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 host fails
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
This is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu (Ubuntu 18.04) is not compatible with older qemu in (Ubuntu 16.04.4).
Machine Type = Boston
---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest with explicit compat mode setting on P8 host due to the reason mentioned on https://bugzilla.linux.ibm.com/show_bug.cgi?id=166667#c22,
<cpu mode='host-model'>
<model>power8</model>
</cpu>
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration from Ubuntu 16.04.4 -> Ubuntu 18.04 and it works as expected,
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [100 %]
4. Perform backward migration of the same guest from Ubuntu 18.04 P9 host -> Ubuntu 16.04.4 P8 host and it is broken,
# virsh -c qemu+ssh://9.40.194.204/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [ 99 %]error: internal error: early end of file from monitor, possible problem: 2018-05-02T12:49:07.324377Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-05-02T12:49:07.332143Z qemu-system-ppc64: load of migration failed: Invalid argument
---uname output---
Ubuntu 18.04 P9 Host:
Kernel
# uname -a
Linux ltc-boston114 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:14:44 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu 1:2.11+dfsg-1ubuntu7 ppc64el fast processor emulator
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
ii qemu-utils 1:2.11+dfsg-1ubuntu7 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu8 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu8 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu8 ppc64el Virtualization daemon
ii libvirt-daemon-system 4.0.0-1ubuntu8 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu8 ppc64el development files for the libvirt library
ii libvirt-doc 4.0.0-1ubuntu8 all documentation for the libvirt library
Ubuntu 16.04.4 P8 Host:
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.25 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.25 ppc64el QEMU utilities
Libvirt:
# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.21 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.21 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.21 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings
Ubuntu 16.04.4 P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-39-generic #44~16.04.1-Ubuntu SMP Thu Apr 5 16:41:53 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Attachment:
1. Libvirtd debug logs: source and destination host
2. Sosreport: source host, destination host and Guest
---Debugger---
A debugger is not configured
Userspace tool common name: qemu
Userspace rpm: qemu-kvm - 1:2.11+dfsg-1ubuntu7
The userspace tool has the following bit modes: ppc64le
Userspace tool obtained from project website: na
== Comment: #5 - Greg Kurz - 2018-05-04 07:21:05 ==
Backward migration had three distinct issues. Fixes are now upstream:
https://git.qemu.org/?p=qemu.git;a=commit;h=bce009645b9f1d59195518e35747c8ea30f985f7
https://git.qemu.org/?p=qemu.git;a=commit;h=aef19c04bf88e0f5f936301e6c29b239e488fbc6
https://git.qemu.org/?p=qemu.git;a=commit;h=0550b1206a91d66051a21441a02c4ff126b531fe
I'll make sure they land into QEMU 2.11.2 and QEMU 2.12.1 as well.
== Comment: #6 - Balamuruhan S <balamuruhans@in.ibm.com> - 2018-06-08 07:49:51 ==
Tested with latest qemu build available for Ubuntu 18.04,
Issue is still observed with Qemu available in Ubuntu 18.04, I think the patchset is not
included in Qemu 1:2.11+dfsg-1ubuntu7.2
Migrating P8 Ubuntu16.04.4 guest from P8 Ubuntu 16.04.4 host -> P9 Ubuntu 18.04 host
# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.193.215/system --live --verbose
Migration: [100 %]
Migrating back P8 Ubuntu 16.04.4 guest from P9 Ubuntu 18.04 host -> P8 Ubuntu 16.04.4 host
# virsh -c qemu+ssh://9.40.193.215/system migrate avocado-vt-vm1-migration qemu+ssh://9.47.70.201/system --live --verbose
root@9.47.70.201's password:
Migration: [100 %]error: internal error: early end of file from monitor, possible problem: 2018-06-08T12:46:40.743395Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-06-08T12:46:40.751718Z qemu-system-ppc64: load of migration failed: Invalid argument
P9 Ubuntu 18.04:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu7.2 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu7.2 ppc64el QEMU utilities
P8 Ubuntu 16.04.4:
Qemu
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.29 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.29 ppc64el QEMU utilities
== Comment: #11 - Greg Kurz - 2018-12-14 04:11:37 ==
According to the QEMU website at https://www.qemu.org/ , QEMU 2.11.2 got released on Jun 27, 2018.
According to the Ubuntu release page at https://wiki.ubuntu.com/Releases , Ubuntu 18.04.1 LTS to released on July 26, 2018.
greg@wslp001:~$ lsb_release -d
Description: Ubuntu 18.04.1 LTS
greg@wslp001:~$ qemu-system-ppc64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.9)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
I don't know why Canonical didn't pick up the latest stable release... and I'm not sure that fixing backward migration meets the criteria for an SRU. |
|
| 2019-02-05 10:09:14 |
Frank Heimes |
qemu (Ubuntu Cosmic): status |
Triaged |
In Progress |
|
| 2019-02-05 10:09:27 |
Frank Heimes |
qemu (Ubuntu Bionic): status |
Triaged |
In Progress |
|
| 2019-02-05 19:29:07 |
Brian Murray |
qemu (Ubuntu Cosmic): status |
In Progress |
Fix Committed |
|
| 2019-02-05 19:29:09 |
Brian Murray |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-02-05 19:29:12 |
Brian Murray |
bug |
|
|
added subscriber SRU Verification |
| 2019-02-05 19:29:16 |
Brian Murray |
tags |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-needed verification-needed-cosmic |
|
| 2019-02-07 09:27:51 |
Christian Ehrhardt |
tags |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-needed verification-needed-cosmic |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done verification-done-cosmic |
|
| 2019-02-14 16:23:53 |
Launchpad Janitor |
qemu (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-02-14 16:24:09 |
Łukasz Zemczak |
removed subscriber Ubuntu Stable Release Updates Team |
|
|
|
| 2019-02-25 12:10:56 |
Łukasz Zemczak |
qemu (Ubuntu Bionic): status |
In Progress |
Fix Committed |
|
| 2019-02-25 12:10:58 |
Łukasz Zemczak |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-02-25 12:11:02 |
Łukasz Zemczak |
tags |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done verification-done-cosmic |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done-cosmic verification-needed verification-needed-bionic |
|
| 2019-02-25 12:40:02 |
Frank Heimes |
ubuntu-power-systems: status |
In Progress |
Fix Committed |
|
| 2019-02-25 14:53:59 |
Christian Ehrhardt |
tags |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done-cosmic verification-needed verification-needed-bionic |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done verification-done-bionic verification-done-cosmic |
|
| 2019-03-04 16:41:31 |
Launchpad Janitor |
qemu (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-03-04 16:53:38 |
Frank Heimes |
ubuntu-power-systems: status |
Fix Committed |
Fix Released |
|
| 2020-01-20 14:20:25 |
bugproxy |
tags |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin--- verification-done verification-done-bionic verification-done-cosmic |
architecture-ppc64le bugnameltc-167401 qemu-19.04 severity-high targetmilestone-inin18041 verification-done verification-done-bionic verification-done-cosmic |
|
