[Ubuntu 18.04] P8 compat mode migration fails for Ubuntu 16.04.4 P8 Guest from Ubuntu 16.04.4 P8 Host -> Ubuntu 18.04 P9 Host

Bug #1763468 reported by bugproxy on 2018-04-12
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
High
Canonical Server Team
qemu (Ubuntu)
High
David Britton

Bug Description

---Problem Description---
P8 compat mode migration fails for Ubuntu 16.04.4 P8 Guest from Ubuntu 16.04.4 P8 Host -> Ubuntu 18.04 P9 Host

We need backport of upstream qemu patch and its dependencies to Ubuntu 18.04 to fix it,

commit 72fdd4de8e5fdc1a6078e000835fb54592a3fe97
Author: Greg Kurz <email address hidden>
Date: Tue Feb 27 16:22:58 2018 +0100

    spapr: register dummy ICPs later

    Some older machine types create more ICPs than needed. We hence
    need to register up to xics_max_server_number() dummy ICPs to
    accomodate the migration of these machine types.

    Recent VSMT rework changed xics_max_server_number() to return

        DIV_ROUND_UP(max_cpus * spapr->vsmt, smp_threads)

    instead of

        DIV_ROUND_UP(max_cpus * kvmppc_smt_threads(), smp_threads);

    The change is okay but it requires spapr->vsmt to be set, which
    isn't the case with the current code. This causes the formula to
    return zero and we don't create dummy ICPs. This breaks migration
    of older guests as reported here:

        https://bugzilla.redhat.com/show_bug.cgi?id=1549087

    The dummy ICP workaround doesn't really have a dependency on XICS
    itself. But it does depend on proper VCPU id numbering and it must
    be applied before creating vCPUs (ie, creating real ICPs). So this
    patch moves the workaround to spapr_init_cpus(), which already
    assumes VSMT to be set.

    Fixes: 72194664c8a1 ("spapr: use spapr->vsmt to compute VCPU ids")
    Reported-by: Lukas Doktor <email address hidden>
    Signed-off-by: Greg Kurz <email address hidden>
    Signed-off-by: David Gibson <email address hidden>

---uname output---

P9 Host Ubuntu 18.04 (Destination)
Kernel:
# uname -a
Linux ltc-boston17 4.15.0-15-generic #16-Ubuntu SMP Wed Apr 4 13:57:51 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20180124.fbe8c52d-0ubuntu2 all PXE boot firmware - ROM images for qemu
ii ipxe-qemu-256k-compat-efi-roms 1.0.0+git-20150424.a25a16d-0ubuntu2 all PXE boot firmware - Compat EFI ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.11+dfsg-1ubuntu6 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.11+dfsg-1ubuntu6 ppc64el QEMU Full virtualization on x86 hardware
ii qemu-slof 20170724+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.11+dfsg-1ubuntu6 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.11+dfsg-1ubuntu6 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.11+dfsg-1ubuntu6 ppc64el QEMU utilities

# dpkg -l | grep libvirt
ii libvirt-bin 4.0.0-1ubuntu7 ppc64el programs for the libvirt library
ii libvirt-clients 4.0.0-1ubuntu7 ppc64el Programs for the libvirt library
ii libvirt-daemon 4.0.0-1ubuntu7 ppc64el Virtualization daemon
ii libvirt-daemon-driver-storage-rbd 4.0.0-1ubuntu7 ppc64el Virtualization daemon RBD storage driver
ii libvirt-daemon-system 4.0.0-1ubuntu7 ppc64el Libvirt daemon configuration files
ii libvirt-dev:ppc64el 4.0.0-1ubuntu7 ppc64el development files for the libvirt library
ii libvirt-glib-1.0-0:ppc64el 1.0.0-1 ppc64el libvirt GLib and GObject mapping library
ii libvirt0:ppc64el 4.0.0-1ubuntu7 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 4.0.0-1 ppc64el libvirt Python bindings

P8 Host Ubuntu 16.04.4 (Source)
Kernel:
# uname -a
Linux pkvmhab006 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:46:55 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Qemu:
# dpkg -l | grep qemu
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1.2 all PXE boot firmware - ROM images for qemu
ii qemu-block-extra:ppc64el 1:2.5+dfsg-5ubuntu10.24 ppc64el extra block backend modules for qemu-system and qemu-utils
ii qemu-kvm 1:2.5+dfsg-5ubuntu10.24 ppc64el QEMU Full virtualization
ii qemu-slof 20151103+dfsg-1ubuntu1.1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.24 ppc64el QEMU full system emulation binaries (common files)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.24 ppc64el QEMU full system emulation binaries (ppc)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.24 ppc64el QEMU utilities

# dpkg -l | grep libvirt
ii libvirt-bin 1.3.1-1ubuntu10.19 ppc64el programs for the libvirt library
ii libvirt-dev:ppc64el 1.3.1-1ubuntu10.19 ppc64el development files for the libvirt library
ii libvirt0:ppc64el 1.3.1-1ubuntu10.19 ppc64el library for interfacing with different virtualization systems
ii python-libvirt 1.3.1-1ubuntu1.1 ppc64el libvirt Python bindings

P8 Guest:
Kernel:
# uname -a
Linux ubuntu 4.13.0-38-generic #43~16.04.1-Ubuntu SMP Wed Mar 14 17:46:55 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux
Machine Type = Boston

---Debugger---
A debugger is not configured

---Steps to Reproduce---
1. Boot healthy P8 16.04.4 Ubuntu guest on P8 host
2. Have P9 Host with 18.04 Ubuntu build and allow iptables to permit migration
3. Perform migration and the error observed,

virsh -c 'qemu:///system' migrate --live --domain avocado-vt-vm1-migration --desturi qemu+ssh://9.40.193.215/system --timeout 60

error: internal error: qemu unexpectedly closed the monitor: on,vhostfd=32 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:c6:c7:c8,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -incoming defer -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on
2018-04-12T11:28:56.425071Z qemu-system-ppc64: Unknown savevm section or instance 'icp/server' 32
2018-04-12T11:28:56.431349Z qemu-system-ppc64: load of migration failed: Invalid argument

Userspace tool common name: Qemu

The userspace tool has the following bit modes: ppc64le

Userspace package: qemu-kvm 1:2.11+dfsg-1ubuntu6

bugproxy (bugproxy) on 2018-04-12
tags: added: architecture-ppc64le bugnameltc-166667 severity-critical targetmilestone-inin1804
Changed in ubuntu:
assignee: nobody → Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage)
affects: ubuntu → qemu (Ubuntu)
Changed in ubuntu-power-systems:
importance: Undecided → Critical
tags: added: triage-g
Changed in ubuntu-power-systems:
importance: Critical → High
assignee: nobody → Canonical Server Team (canonical-server)

I tested P8->P8 but couldn't P8->P9 lacking the HW in the same environment.
Thanks for spotting that issue, taking a look at backportability of this ...

Changed in qemu (Ubuntu):
status: New → Triaged
Frank Heimes (fheimes) on 2018-04-13
Changed in ubuntu-power-systems:
status: New → Triaged

The simply said "its dependencies" seems to be a dependency hell.
  72fdd4de8e5 spapr: register dummy ICPs later
referred to fix patch:
  72194664c8a1 spapr: use spapr->vsmt to compute VCPU ids
is not in Qemu 2.11, nor is the one this fixes
  8904e5a75005 spapr: Adjust default VSMT value for better migration compatibility

Without testing it seems we would need:
- ? marks part of the series I'm unsure it is a hard requirement but it seems entangled enough vsmt/vcpuid to consider it as well to not break functionality :-/
- f#<num> marks which other change it fixes (implicit depends)
- r#<num> marks which other change depends on it

#1 ? r#2 1a5008fc spapr: harden code that depends on VSMT
#2 f#7 72fdd4de spapr: register dummy ICPs later
#3 ? f#4 b1a568c1 spapr: fix missing CPU core nodes in DT when running with TCG
#4 ? 5d0fb150 spapr: consolidate the VCPU id numbering logic in a single place
#5 ? 14bb4486 rename spapr_vcpu_id() to spapr_get_vcpu_id()
#6 ? 648edb64 spapr: move VCPU calculation to core machine code
#7 f#9 72194664 spapr: use spapr->vsmt to compute VCPU ids
#8 f#9 4ad64cbd spapr: set vsmt to MAX(8, smp_threads)
#9 f#2.11 8904e5a7 spapr: Adjust default VSMT value for better migration compatibility
#10 r#9 1f20f2e0 spapr: Allow some cases where we can't set VSMT mode in the kernel

#1 has too much noise and isn't a hard requirement, so I skipped it
There also was some noise for:
- max_vthreads being renamed
- on #7 for threads[0]
- on #6 for POWERPC_CPU
- on #6 for spapr_caps_pre_load
but that was ok.

Overall backport in packaged format would be:
- https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=416b00af3a8bbfc26094c4a3eba0bce09888af6c
- https://git.launchpad.net/~paelzer/ubuntu/+source/qemu/commit/?id=0c93d6500d13e9eeefcf8797e0f05e0225bf9438

I made it part of the ppa that was open for bug 1762854 - so you can test both at once.
=> https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3233

So it seems doable, but it is quite a huge change still.
9 patches with overall:
 hw/ppc/spapr.c | 160 ++++++++++++++++++++++++++++++------------------
 hw/ppc/spapr_cpu_core.c | 9 --
 include/hw/ppc/spapr.h | 3

OTOH it is ppc64 only so it is your (=IBM) call to make, do you want to use the backport or do you think there is a much smaller fix (changing 1-5 lines somewhere) for qemu 2.11 available to "fix just the issue"?
This is a tradeoff, if it becomes too complex then the backport is clearly preferred, but if there would be a way to just handle this more comprehensively without introducing all the new VSMT handling first let me know.

TL;DR I'd ask you to:
- evaluate if a (much) simpler fix would be available
- if not please review the backport to fit your needs
- also then verify the ppa provided so that we can go forward

Setting status to incomplete as I wait for feedback on that.

Changed in qemu (Ubuntu):
status: Triaged → Incomplete
Changed in ubuntu-power-systems:
status: Triaged → Incomplete

------- Comment From <email address hidden> 2018-04-13 10:57 EDT-------
(In reply to comment #6)
> TL;DR I'd ask you to:
> - evaluate if a (much) simpler fix would be available

Yeah, I believe we can come up with something slightly simpler.

The kernel for 18.04 is 4.15, hence KVM supports setting the VSMT, and we
can drop all the patches that deal with kvmppc_smt_threads() != spapr->vsmt.

This means the following patches can be dropped from the backport:

lp-1763468-1-spapr-Allow-some-cases-where-we-can-t-set-VSMT-mode-.patch
lp-1763468-4-spapr-use-spapr-vsmt-to-compute-VCPU-ids.patch

These ones are cleanup and can be dropped:

lp-1763468-5-spapr-move-VCPU-calculation-to-core-machine-code.patch
lp-1763468-6-spapr-rename-spapr_vcpu_id-to-spapr_get_vcpu_id.patch
lp-1763468-7-spapr-consolidate-the-VCPU-id-numbering-logic-in-a-s.patch

This one fixes a regression from the above cleanup and can be dropped:

lp-1763468-8-spapr-fix-missing-CPU-core-nodes-in-DT-when-running-.patch

But we still need to adjust the default VSMT value, so we want to keep these:

lp-1763468-2-spapr-Adjust-default-VSMT-value-for-better-migration.patch
lp-1763468-3-spapr-set-vsmt-to-MAX-8-smp_threads.patch

and we also need to register dummy ICPs later, to "fix just the issue". This requires
to adapt lp-1763468-9-spapr-register-dummy-ICPs-later.patch to the fact that
xics_max_server_number() doesn't take arguments. The following does the trick:

perl -i -p -e 's/xics_max_server_number\(spapr\)/xics_max_server_number()/' \
lp-1763468-9-spapr-register-dummy-ICPs-later.patch

So you end up with 3 patches and a nicer diffstat:

hw/ppc/spapr.c | 35 ++++++++++++++++++++---------------
1 file changed, 20 insertions(+), 15 deletions(-)

Is it enough for you to go forward ?

Manoj Iyer (manjo) on 2018-04-16
Changed in qemu (Ubuntu):
status: Incomplete → Triaged
Changed in ubuntu-power-systems:
status: Incomplete → Triaged
Manoj Iyer (manjo) on 2018-04-16
Changed in qemu (Ubuntu):
importance: Undecided → High
assignee: Ubuntu on IBM Power Systems Bug Triage (ubuntu-power-triage) → David Britton (davidpbritton)
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-04-16 15:39 EDT-------
FYI, I could talk with Michael Roth, maintainer of QEMU stable: QEMU 2.11.2 will ship the 3 patches mentioned in comment #7.

Thanks Greg for taking a look with me.
We unfortunately can't rely on kernel 4.15 due to the fact that the virt stack in 18.04 will be made available to Xenial via the Ubuntu Cloud Archive [1].

So this could run on hosts as much back as 4.4
With that said we would still need patches -1 and -4.

You said -5, -6 and -7 (plus -8 as fix) are only cleanup so those we might drop.
Due to the dependencies outlined before I kept them in the series, but I'm happy to follow your guidance on that.

And with -1, -2, -3, -4, - 9 there is not even noise on xics_max_server_number.

I made a respin of the build in the ppa [2] with that.
Please verify if that suits your needs for:
- this bug 1763468
- other current ppc bug 1762854

[1]: https://wiki.ubuntu.com/OpenStack/CloudArchive
[2]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3233

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-04-17 10:22 EDT-------
(In reply to comment #9)
>
> And with -1, -2, -3, -4, - 9 there is not even noise on
> xics_max_server_number.
>

Patch list was reviewed and tested internally. You can go ahead with this fix.

Thanks for the verify, pushed to Bionic-unapproved

Got P9 HW myself:
current qemu:
a) P8->P9 worked
b) P8->P9->P8 failed (silently)

proposed qemu for this bug:
a) P8->P9 worked
b) P8->P9->P8 worked

P9->P8 also working if one sets the right compat, like for example
Default on P9 guest:
cpu : POWER9 (architected), altivec supported
revision : 2.2 (pvr 004e 1202)
Set compat:
 <cpu mode='host-model' check='partial'>
   <model fallback='allow'>power8</model>
 </cpu>
Then is is in the guest:
cpu : POWER8 (architected), altivec supported
revision : 2.2 (pvr 004e 1202)
And this migrates P9->P8

+1 on confidence of the proposed upload

Changed in ubuntu-power-systems:
status: Triaged → In Progress
Changed in qemu (Ubuntu):
status: Triaged → In Progress
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7

---------------
qemu (1:2.11+dfsg-1ubuntu7) bionic; urgency=medium

  * d/p/ubuntu/lp-1762854-*: fix issue with SCSI-2 devices denying Protection
    information (LP: #1762854).
  * d/p/ubuntu/lp-1763468-*: fix VSMT handling to fix ppc64el P8/P9 migration
    (LP: #1763468).

 -- Christian Ehrhardt <email address hidden> Wed, 11 Apr 2018 07:46:18 +0200

Changed in qemu (Ubuntu):
status: In Progress → Fix Released
Changed in ubuntu-power-systems:
status: In Progress → Fix Released
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-04-30 02:34 EDT-------
(In reply to comment #14)
> This bug was fixed in the package qemu - 1:2.11+dfsg-1ubuntu7
>
> ---------------
> qemu (1:2.11+dfsg-1ubuntu7) bionic; urgency=medium
>
> * d/p/ubuntu/lp-1762854-*: fix issue with SCSI-2 devices denying Protection
> information (LP: #1762854).
> * d/p/ubuntu/lp-1763468-*: fix VSMT handling to fix ppc64el P8/P9 migration
> (LP: #1763468).
>
> -- Christian Ehrhardt <email address hidden> Wed, 11 Apr 2018
> 07:46:18 +0200

I observe issue with loading cpu state in target now with qemu (1:2.11+dfsg-1ubuntu7),

# virsh migrate avocado-vt-vm1-migration qemu+ssh://9.40.194.204/system --live --verbose
Migration: [ 99 %]error: internal error: qemu unexpectedly closed the monitor: 2018-04-30T06:29:00.728297Z qemu-system-ppc64: error while loading state for instance 0x0 of device 'cpu'
2018-04-30T06:29:00.735891Z qemu-system-ppc64: load of migration failed: Operation not permitted

System configuration:

Ubuntu 18.04 P9 Host:
Kernel - 4.15.0-20-generic
Qemu - 1:2.11+dfsg-1ubuntu7
SLOF - 20170724+dfsg-1ubuntu1
Libvirt - 4.0.0-1ubuntu8

Ubuntu 16.04.4 P8 Host:
Kernel - 4.13.0-39-generic
Qemu - 1:2.5+dfsg-5ubuntu10.25
SLOF - 20151103+dfsg-1ubuntu1.1
Libvirt - 1.3.1-1ubuntu10.21

Ubuntu 16.04.4 P8 Guest:
Kernel - 4.13.0-39-generic

-- Bala

Hi Bala,
interesting, but not 100% the same case IMHO.

The former fix and discussion was about 18.04 on P8 to 18.04 on P9 and it worked fine after the fixes that were pointed out were integrated.

I'd ask you to file a new bug (internally for you and your PPC dev's as well as with launchpad for us) to discuss potential needs for 16.04 on P8 to 18.04 on P9. Only then I think tracking and discussions will go well.

The old title defies my understanding of this case thou, never the less all changes so far were to 18.04.
And on confirmation it was 18.04 broken -> 18.04 working after the fix.
I'll set up a few systems to try if they are free atm.
But a new bug would still be appreciated.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-04-30 03:16 EDT-------
(In reply to comment #16)
> Hi Bala,
> interesting, but not 100% the same case IMHO.
>
> The former fix and discussion was about 18.04 on P8 to 18.04 on P9 and it
> worked fine after the fixes that were pointed out were integrated.

This Bug was originally raised for issue in migrating Ubuntu 16.04.4 P8 Guest on Ubuntu 16.04.4 P8 Host -> Ubuntu 18.04 P9 Host.

>
> I'd ask you to file a new bug (internally for you and your PPC dev's as well
> as with launchpad for us) to discuss potential needs for 16.04 on P8 to
> 18.04 on P9. Only then I think tracking and discussions will go well.

I think the potential reason is that, it is the basic usecase for a customer currently having Ubuntu 16.04.4 P8 Guest on Ubuntu 16.04.4 P8 Host, wants to move the guest to P9 Host or in cloud if the migration takes place in automated way after plugging in P9 servers in racks migration should be seamless.

-- Bala

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-04-30 04:02 EDT-------
(In reply to comment #18)
> The old title defies my understanding of this case thou, never the less all
> changes so far were to 18.04.
> And on confirmation it was 18.04 broken -> 18.04 working after the fix.
> I'll set up a few systems to try if they are free atm.
> But a new bug would still be appreciated.

I second Bala's statements: the list of fixes was decided with 16.04.4 -> 18.04 migration in mind... I'm now working with Bala to investigate.

Ok, if that was what you had in mind when selecting patches fine.
Thanks for taking a look.

I'll for tracking eventually fork a bug number for the remaining issue on this then, but this can wait until it is discussed and pre-tested here.

I wanted to take a look at this again, but TBH at the moment all I've got was crashing my P9 system three times. I'll start a mail thread on it and keep this bug clean of it for now.

While resolving that I hope that Bala/Greg will find something about the issue here on their side.

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-05-02 08:34 EDT-------
After working with Greg, we found that migration of Ubuntu 16.04.4 guest from Ubuntu 16.04.4 P8 Host to ubuntu 18.04 P9 host works but we need to explicitly boot the guest in source with compat settings even on P8 host, ie

<cpu mode='host-model'>
<model>power8</model>
</cpu>

As Greg stated, this is because Ubuntu 16.04.4 use older qemu (Qemu 2.5) that boots the guest in raw mode where as Ubuntu 18.04 use newer Qemu (Qemu 2.11) that boots the guest in architected mode, which means if the guest runs in raw mode, then migration is only supported between identical hosts and if it run in architected mode, the guest can be migrated to any host that also supports this architected mode.

In our case we try to migrate from raw mode to architected mode that's why we have to give explicit compat setting is source so that respective qemu command will be formed with compat setting in target for migration to start. This is limitation we have from past and Greg would be working on it to fix.

We can safely close this bug but with a proper documentation to make sure this limitation is known to the user.

But still backward migration is broken, migrating back Ubuntu 16.04.4 guest from Ubuntu 18.04 P9 Host -> Ubuntu 16.04.4 P8 Host is broken because cpu model QEMU 2.5 (-cpu host,compat=power8) and QEMU 2.11 (-machine max-cpu-compat=power8) known for latest qemu is not compatible with older qemu. So this is a new issue with different root cause for which I would raise a separate bug.

Thanks Greg.

-- Bala

bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2018-05-23 02:33 EDT-------
Hello Canonical,
FYI we have now closed this bug by documenting the limitation as below. Thanks!

https://wiki.ubuntu.com/ppc64el/uKVM#Limitation_in_migrating_POWER8_guest_from_POWER8_host_to_POWER9_host

Thanks Iranna for the update, the wiki entry LGTM

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers