ubuntu-kvm-unit test failed with sprs test on ppc64le

[Po-Hsu Lin]

[Impact]

 * Certain Guest features (SPRS) of power8 isa are not working

 * This is part of the ongoing LTS HW exploitation and a fixup to some
   verification tests at the same time.

[Test Case]

 * 1. deploy xenial + HWE kernel on a ppc64el box
   2. sudo apt-get install qemu-kvm -y
   3. git clone --depth=1 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
   4. cd kvm-unit-tests
   5. ./configure --endian=little; make
   6. sudo ppc64_cpu --smt=off
   7. sudo ./run_tests.sh -v

[Regression Potential]

 * The changes are ppc only and provided by IBM so the potential risk is
   retained to the owner of the Area.
   Further if these registers are not used they do nothing, so in most
   cases it will be a nop.
   OTOH the changes are rather huge (in terms of LOC) so there is some
   risk left.

[Other Info]

 * n/a

---

The sprs test has failed on ppc64le
This issue can be spotted on Zesty and Xenial. There is no history for this test on Artful

$ sudo /bin/bash -c "MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'"
run_migration timeout -k 1s --foreground 90s /usr/bin/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/sprs.elf -smp 1 -append -w # -initrd /tmp/tmp.RUTL6E2JSi
Settings SPRs to 0xcafefacec0debabe...
Now migrate the VM, then press a key or send NMI...
Checking SPRs...
PASS: SPR 3: 0x0000000000debabe <==> 0x0000000000debabe
PASS: SPR 13: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 17: 0x0000000000debabe <==> 0x0000000000debabe
PASS: SPR 18: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 19: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 29: 0xcafefacec0debabe <==> 0xcafefacec0debabe
FAIL: SPR 61: 0x4054504440541014 <==> 000000000000000000
PASS: SPR 153: 0x0a00000000000084 <==> 0x0a00000000000084
PASS: SPR 157: 0xcfffffcfc0cfffff <==> 0xcfffffcfc0cfffff
FAIL: SPR 159: 0x00000000c0debabe <==> 000000000000000000
PASS: SPR 256: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 259: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 274: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 275: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 769: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 770: 0xcafefacec006babe <==> 0xcafefacec006babe
PASS: SPR 771: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 772: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 773: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 774: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 775: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 776: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 779: 0x00000000fa8b1afe <==> 0x00000000fa8b1afe
PASS: SPR 780: 0xcafefacec0debabc <==> 0xcafefacec0debabc
PASS: SPR 781: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 782: 0xcafefacec0debabe <==> 0xcafefacec0debabe
FAIL: SPR 784: 0xcafefacec0debabe <==> 000000000000000000
PASS: SPR 785: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 786: 0xcafefacec006babe <==> 0xcafefacec006babe
PASS: SPR 787: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 788: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 789: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 790: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 791: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 792: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 795: 0x00000000fa8b1afe <==> 0x00000000fa8b1afe
PASS: SPR 796: 0xcafefacec0debabc <==> 0xcafefacec0debabc
PASS: SPR 797: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 798: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 800: 0x80000000c0000000 <==> 0x80000000c0000000
PASS: SPR 801: 0x0000000080000000 <==> 0x0000000080000000
PASS: SPR 802: 0x80000000c0000000 <==> 0x80000000c0000000
PASS: SPR 803: 0x0000000080000000 <==> 0x0000000080000000
PASS: SPR 804: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 805: 0xcafefacec0debabc <==> 0xcafefacec0debabc
PASS: SPR 806: 0x80000000c0000000 <==> 0x80000000c0000000
FAIL: SPR 815: 0xcafefacec0debabe <==> 000000000000000000
SUMMARY: 47 tests, 4 unexpected failures

EXIT: STATUS=3

$ qemu-system-ppc --version
QEMU emulator version 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.16), Copyright (c) 2003-2008 Fabrice Bellard

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.10.0-37-generic 4.10.0-37.41~16.04.1
ProcVersionSignature: User Name 4.10.0-37.41~16.04.1-generic 4.10.17
Uname: Linux 4.10.0-37-generic ppc64le
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: ppc64el
Date: Mon Oct 16 09:16:08 2017
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcLoadAvg: 0.02 0.14 0.17 1/1464 15093
ProcLocks:
 1: POSIX ADVISORY WRITE 1604 00:14:19651 0 EOF
 2: POSIX ADVISORY WRITE 2937 00:14:55331 0 EOF
 3: FLOCK ADVISORY WRITE 3046 00:14:67623 0 EOF
 4: POSIX ADVISORY WRITE 3124 00:14:57398 0 EOF
 5: POSIX ADVISORY WRITE 3047 00:14:87068 0 EOF
ProcSwaps:
 Filename Type Size Used Priority
 /swap.img file 8388544 0 -1
ProcVersion: Linux version 4.10.0-37-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (User Name/IBM 5.4.0-6ubuntu1~16.04.4) ) #41~16.04.1-User Name SMP Fri Oct 6 22:44:24 UTC 2017
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)
cpu_cores: Number of cores present = 20
cpu_coreson: Number of cores online = 20
cpu_smt: SMT is off

[Po-Hsu Lin (cypressyew)]

qemu-2.7.1 built from source still got this issue.
But it can be fixed by using qemu-2.10.0 built from source.

# export QEMU=/home/ubuntu/qemu-2.10.0/ppc64-softmmu/qemu-system-ppc64
# MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
run_migration timeout -k 1s --foreground 90s /home/ubuntu/qemu-2.10.0/ppc64-softmmu/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/sprs.elf -smp 1 -append -w # -initrd /tmp/tmp.dKAYkHnAZh
Settings SPRs to 0xcafefacec0debabe...
Now migrate the VM, then press a key or send NMI...

Checking SPRs...
PASS: SPR 3: 0x0000000000debabe <==> 0x0000000000debabe
PASS: SPR 13: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 17: 0x0000000000debabe <==> 0x0000000000debabe
PASS: SPR 18: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 19: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 29: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 61: 0x4054504440541014 <==> 0x4054504440541014
PASS: SPR 153: 0x0a00000000000084 <==> 0x0a00000000000084
PASS: SPR 157: 0xcfffffcfc0cfffff <==> 0xcfffffcfc0cfffff
PASS: SPR 159: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 256: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 259: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 274: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 275: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 769: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 770: 0xcafefacec006babe <==> 0xcafefacec006babe
PASS: SPR 771: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 772: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 773: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 774: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 775: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 776: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 779: 0x00000000fa8b1afe <==> 0x00000000fa8b1afe
PASS: SPR 780: 0xcafefacec0debabc <==> 0xcafefacec0debabc
PASS: SPR 781: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 782: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 784: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 785: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 786: 0xcafefacec006babe <==> 0xcafefacec006babe
PASS: SPR 787: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 788: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 789: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 790: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 791: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 792: 0x00000000c0debabe <==> 0x00000000c0debabe
PASS: SPR 795: 0x00000000fa8b1afe <==> 0x00000000fa8b1afe
PASS: SPR 796: 0xcafefacec0debabc <==> 0xcafefacec0debabc
PASS: SPR 797: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 798: 0xcafefacec0debabe <==> 0xcafefacec0debabe
PASS: SPR 800: 0x80000000c0000000 <==> 0x80000000c0000000
PASS: SPR 801: 0x0000000080000000 <==> 0x0000000080000000
PASS: SPR 802: 0x80000000c0000000 <==> 0x80000000c0000000
PASS: SPR 803: 0x0000000080000000 <==> 0x0000000080000000
PASS: SPR 804: 0xcafefacec0debabe <==> 0xcafefacec0d...

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1723904

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Christian Ehrhardt  (paelzer)]

As with bug 1712803 please outline the steps to get the test elf image and then we should mirror to IBM so they can check which ppc changes would be needed. Based on knowing that we can then decide to SRU or not.

[Po-Hsu Lin (cypressyew)]

Steps to reproduce:
1. deploy xenial + HWE kernel on a ppc64el box
2. sudo apt-get install qemu-kvm -y
3. git clone --depth=1 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
4. cd kvm-unit-tests
5. ./configure --endian=little; make
6. sudo ppc64_cpu --smt=off
7. sudo ./run_tests -v

From here you will see it's using the elf file from kvm-unit-tests/powerpc and failed with 3 tests:
 - spapr_hcall (bug 1712803)
 - emulator (bug 1723914)
 - sprs (bug 1723904)

To run it with a newer qemu built from source:
1. wget http://download.qemu-project.org/qemu-2.10.0.tar.xz
2. tar -xf qemu-2.10.0.tar.xz
3. cd qemu-2.10.0
4. ./configure; make
5. cd ~/kvm-unit-tests
6. sudo su
7. export QEMU=/home/ubuntu/qemu-2.10.0/ppc64-softmmu/qemu-system-ppc64
8. ./run_test -v

[bugproxy (bugproxy)]

Default Comment by Bridge

[Po-Hsu Lin (cypressyew)]

For Zesty with the kvm-unit-test code from upstream, it will hang with this test on ppc64le now, bug 1740017

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2018-03-29 16:23 EDT-------
Some SPR registers were not yet defined in qemu 2.5 and therefore are failing this test. The problem is gone from qemu 2.6. Sending the backport patches to Canonical team.

[Yasmin Beatriz Alves da Silva (yasmins)]

Hi @paelzer,

Here are the backport patches. Let me know if there are any problems.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "canonical_sprs_backport.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

[Christian Ehrhardt  (paelzer)]

I plan to combine this with the fix for bug 1723914 which would allow you to only run one verification for both.

[Christian Ehrhardt  (paelzer)]

Since the fix is in 2.9 only Xenial is considered for the SRU as >=Artful is good

[Christian Ehrhardt  (paelzer)]

SRU Template prepared

Once ppa is built:
- Regression tests
- per case tests

Note: test build in ppa at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3224

[Christian Ehrhardt  (paelzer)]

Tested the new build with the outlined testcase:
  MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
  PASS sprs

[Christian Ehrhardt  (paelzer)]

Regression tests started on all arches

[Christian Ehrhardt  (paelzer)]

Regression tests good as well, moving it to x-unapproved for the SRU Team to consider.
=> qemu_2.5+dfsg-5ubuntu10.25_source.changes

[Łukasz Zemczak (sil2100)]

An upload of qemu to xenial-proposed has been rejected from the upload queue for the following reason: "What is the reason that the previous security update".

[Łukasz Zemczak (sil2100)]

Pressed enter too soon so my rejection message was cut:
"What is the reason that the previous security update's patched have been removed/disabled? Do they introduce any regressions that we didn't know about? Or maybe the new patches also address the CVEs ? If so, this should be mentioned in the changelog and/or forwarded to the security team. New SRUs should not overwrite -security changes - all pockets should have security fixes."

[Łukasz Zemczak (sil2100)]

Hello Po-Hsu, or anyone else affected,

Accepted qemu into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu10.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Christian Ehrhardt  (paelzer)]

same test as the other bug - so same update

# sudo ./run_tests.sh -v
[...]
TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
FAIL emulator
SKIP h_cede_tm (test marked as manual run only)
MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
FAIL sprs

Upgrade to fix (no issues on upgrade)

# sudo ./run_tests.sh -v
[...]
TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
PASS emulator
SKIP h_cede_tm (test marked as manual run only)
MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
PASS sprs

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.25

---------------
qemu (1:2.5+dfsg-5ubuntu10.25) xenial; urgency=medium

  [ Murilo Opsfelder Araújo ]
  * debian/patches/ubuntu/lp-1723914-ppc64-set-msr-register.patch:
    properly set MSR register for ppc64 targets (LP: #1723914)

  [ Yasmin Beatriz Alves da Silva ]
  * debian/patches/ubuntu/lp-1723904-sprs-backport.patch:
    Fix handling of Power8 special purpose registers (LP: #1723904)

 -- Christian Ehrhardt <email address hidden> Wed, 04 Apr 2018 12:46:15 +0200

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.