ubuntu-kvm-unit test failed with emulator test on ppc64le

Bug #1723914 reported by Po-Hsu Lin
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
The Ubuntu-power-systems project
Fix Released
Medium
Unassigned
linux (Ubuntu)
Invalid
Undecided
Unassigned
Xenial
Invalid
Undecided
Unassigned
Artful
Invalid
Undecided
Unassigned
qemu (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Artful
Fix Released
Undecided
Unassigned

Bug Description

[Impact]

 * on start of qemu-system-ppc64 MSR_SF is not set correctly
   So it doesn't start 64bit as it should

 * Todays guests usually work around this and are fine, but it is
   incorrect and might be an issue down the road. Furthermore this is also
   a fixup to some verification tests at the same time.

[Test Case]

 * 1. deploy xenial + HWE kernel on a ppc64el box
   2. sudo apt-get install qemu-kvm -y
   3. git clone --depth=1 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
   4. cd kvm-unit-tests
   5. ./configure --endian=little; make
   6. sudo ppc64_cpu --smt=off
   7. sudo ./run_tests.sh -v

[Regression Potential]

 * The changes are ppc only and provided by IBM so the potential risk is
   retained to the owner of the Area.
   Further the change is very small, so it has a very low amount of being
   misunderstood.

[Other Info]

 * n/a

---

Similar to bug 1723904, bug 1712803.
The failed "emulator" test from kvm-unit-test will pass with qemu-2.7.1 built from source.

ubuntu@modoc:~/kvm-unit-tests$ sudo /bin/bash -c "TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
> "
timeout -k 1s --foreground 90s /usr/bin/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/emulator.elf -smp 1 # -initrd /tmp/tmp.BJ2sE7V2Jc
FAIL: emulator: 64bit: detected
PASS: emulator: invalid: exception
PASS: emulator: lswx: alignment
PASS: emulator: lswi: alignment
SUMMARY: 4 tests, 1 unexpected failures

EXIT: STATUS=3

ProblemType: Bug
DistroRelease: Ubuntu 16.04
Package: linux-image-4.10.0-37-generic 4.10.0-37.41~16.04.1
ProcVersionSignature: User Name 4.10.0-37.41~16.04.1-generic 4.10.17
Uname: Linux 4.10.0-37-generic ppc64le
ApportVersion: 2.20.1-0ubuntu2.10
Architecture: ppc64el
Date: Mon Oct 16 10:10:46 2017
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcLoadAvg: 0.00 0.00 0.57 1/1457 83356
ProcLocks:
 1: POSIX ADVISORY WRITE 1604 00:14:19651 0 EOF
 2: POSIX ADVISORY WRITE 2937 00:14:55331 0 EOF
 3: FLOCK ADVISORY WRITE 3046 00:14:67623 0 EOF
 4: POSIX ADVISORY WRITE 3124 00:14:57398 0 EOF
 5: POSIX ADVISORY WRITE 3047 00:14:87068 0 EOF
ProcSwaps:
 Filename Type Size Used Priority
 /swap.img file 8388544 0 -1
ProcVersion: Linux version 4.10.0-37-generic (buildd@bos01-ppc64el-026) (gcc version 5.4.0 20160609 (User Name/IBM 5.4.0-6ubuntu1~16.04.4) ) #41~16.04.1-User Name SMP Fri Oct 6 22:44:24 UTC 2017
SourcePackage: linux-hwe
UpgradeStatus: No upgrade log present (probably fresh install)
cpu_cores: Number of cores present = 20
cpu_coreson: Number of cores online = 20
cpu_smt: SMT is off

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

With qemu-2.7.1 built from source:
# export QEMU=/home/ubuntu/qemu-2.7.1/ppc64-softmmu/qemu-system-ppc64
# TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
timeout -k 1s --foreground 90s /home/ubuntu/qemu-2.7.1/ppc64-softmmu/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/emulator.elf -smp 1 # -initrd /tmp/tmp.2wiGOtABBN
PASS: emulator: 64bit: detected
PASS: emulator: invalid: exception
PASS: emulator: lswx: alignment
PASS: emulator: lswi: alignment
SUMMARY: 4 tests

EXIT: STATUS=1

affects: linux-hwe (Ubuntu) → linux (Ubuntu)
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Steps to reproduce:
1. deploy xenial + HWE kernel on a ppc64el box
2. sudo apt-get install qemu-kvm -y
3. git clone --depth=1 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
4. cd kvm-unit-tests
5. ./configure --endian=little; make
6. sudo ppc64_cpu --smt=off
7. sudo ./run_tests -v

From here you will see it's using the elf file from kvm-unit-tests/powerpc and failed with 3 tests:
 - spapr_hcall (bug 1712803)
 - emulator (bug 1723914)
 - sprs (bug 1723904)

To run it with a newer qemu built from source:
1. wget http://download.qemu-project.org/qemu-2.10.0.tar.xz
2. tar -xf qemu-2.10.0.tar.xz
3. cd qemu-2.10.0
4. ./configure; make
5. cd ~/kvm-unit-tests
6. sudo su
7. export QEMU=/home/ubuntu/qemu-2.10.0/ppc64-softmmu/qemu-system-ppc64
8. ./run_test -v

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1723914

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
bugproxy (bugproxy)
tags: added: architecture-ppc64le bugnameltc-160548 severity-medium targetmilestone-inin---
bugproxy (bugproxy)
tags: added: targetmilestone-inin16043
removed: targetmilestone-inin---
Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

This test has passed on 4.13 Artful, need to check on other releases. (Xenial)

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

It's still failing on Xenial, let keep this bug open.

# TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
timeout -k 1s --foreground 90s /usr/bin/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/emulator.elf -smp 1 # -initrd /tmp/tmp.1U5S1gr1oT
FAIL: emulator: 64bit: detected
PASS: emulator: invalid: exception
PASS: emulator: lswx: alignment
PASS: emulator: lswi: alignment
SUMMARY: 4 tests, 1 unexpected failures

EXIT: STATUS=3
Rroot@modoc:/home/ubuntu/kvm-unit-tests# uname -a
Linux modoc 4.13.0-36-generic #40~16.04.1-Ubuntu SMP Fri Feb 16 23:27:02 UTC 2018 ppc64le ppc64le ppc64le GNU/Linux

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Po-Hsu,
are you still debugging this - or do you want to ask this to be mirrored to IBM?
It could again be something that is only working in latter versions - and depending on the change we might (or not) backport the fix.

Revision history for this message
Po-Hsu Lin (cypressyew) wrote :

Hello Christian,
No I'm not debugging this. Not sure what's our policy for this kind of issue, do we always mirror them to IBM?

Came back on this because I saw the qemu update in bug 1712803, tracking bugs found with regression-testing is a part of the SRU testing process.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I think there is no general "mirror all" policy.
It just seems to me that this is requiring an architecture expert which would be at IBM.

I'll subscribe manoj/jfh and let them decide.

Revision history for this message
Manoj Iyer (manjo) wrote : could you pls reverse mirror lp #1723914

Barry/Breno,

Could you please reverse mirror

ubuntu-kvm-unit test failed with emulator test on ppc64le
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1723914

Thanks
--
============================
Manoj Iyer
Ubuntu/Canonical
ARM Servers - Cloud
============================

Revision history for this message
Murilo Opsfelder Araújo (mopsfelder) wrote :

Pior to commit 8b9f2118ca40b1de72d8f75b59a5fb4d347a69f7 ("ppc64: set MSR_SF bit"), qemu-system-ppc64 was not setting the 64-bit mode bit in MSR.

Applying it on top of qemu 2.5 from Ubuntu 16.04.04, makes kvm-unit-tests happy.

muriloo@jaspion1:~/sources/ubuntu$ dpkg-source -x qemu_2.5+dfsg-5ubuntu10.24.dsc
muriloo@jaspion1:~/sources/ubuntu$ cd qemu-2.5+dfsg/
muriloo@jaspion1:~/sources/ubuntu/qemu-2.5+dfsg$ patch -p1 < ~/sources/qemu/0001-ppc64-set-MSR_SF-bit.patch
patching file target-ppc/translate_init.c
Hunk #1 succeeded at 9581 (offset -122 lines).
muriloo@jaspion1:~/sources/ubuntu/qemu-2.5+dfsg$ ./configure --target-list=ppc64-softmmu && make -j$(nproc)
muriloo@jaspion1:~/sources/ubuntu/qemu-2.5+dfsg$ cd ~/sources/kvm-unit-tests/build/
muriloo@jaspion1:~/sources/kvm-unit-tests/build$ ../configure --endian=little && make -j$(nproc)
muriloo@jaspion1:~/sources/kvm-unit-tests/build$ sudo QEMU=/home/muriloo/sources/ubuntu/qemu-2.5+dfsg/ppc64-softmmu/qemu-system-ppc64 TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
[sudo] password for muriloo:
timeout -k 1s --foreground 90s /home/muriloo/sources/ubuntu/qemu-2.5+dfsg/ppc64-softmmu/qemu-system-ppc64 -nodefaults -machine pseries,accel=kvm -bios powerpc/boot_rom.bin -display none -serial stdio -kernel powerpc/emulator.elf -smp 1 # -initrd /tmp/tmp.Tanwa3UNM4
PASS: emulator: 64bit: detected
PASS: emulator: invalid: exception
PASS: emulator: lswx: alignment
PASS: emulator: lswi: alignment
SUMMARY: 4 tests

EXIT: STATUS=1

bugproxy (bugproxy)
tags: added: targetmilestone-inin16045
removed: targetmilestone-inin16043
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: New → Confirmed
importance: Undecided → Medium
Revision history for this message
Manoj Iyer (manjo) wrote :

mopsfelder, could you please provide us a debdiff for xenial?

Revision history for this message
Murilo Opsfelder Araújo (mopsfelder) wrote :
Revision history for this message
Murilo Opsfelder Araújo (mopsfelder) wrote :

Manoj, Christian,

Please refer to the attached debdiff (it was my first debdiff, so it might contain errors :)).

Let me know if you need anything else.

Cheers
Murilo

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "debdiff for Xenial qemu" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

It generally looks good, there are some minor details but I can fix those up when preparing the upload.

In this case it isn't much more than identifying the right upstream change, but thanks a lot as this is often the most important part!

I plan to combine this with the fix for bug 1723904 which would allow you to only run one verification for both.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Since this fix is in 2.6 is is in >=Yakkety, so nonly considering Xenial for the SRU

Changed in qemu (Ubuntu Artful):
status: New → Fix Released
Changed in qemu (Ubuntu Xenial):
status: New → Triaged
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Confirmed → In Progress
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

SRU Template prepared

Once ppa is built:
- Regression tests
- per case tests

Note: test build in ppa at https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/3224

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Tested the new build with the outlined testcase:
  TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
  PASS emulator

description: updated
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Regression tests started on all arches

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Regression tests good as well, moving it to x-unapproved for the SRU Team to consider.
=> qemu_2.5+dfsg-5ubuntu10.25_source.changes

Changed in qemu (Ubuntu Xenial):
status: Triaged → In Progress
Changed in qemu (Ubuntu):
status: New → Fix Released
Frank Heimes (fheimes)
tags: added: triage-g
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Proposed package upload rejected

An upload of qemu to xenial-proposed has been rejected from the upload queue for the following reason: "What is the reason that the previous security update".

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Pressed enter too soon so my rejection message was cut:
"What is the reason that the previous security update's patched have been removed/disabled? Do they introduce any regressions that we didn't know about? Or maybe the new patches also address the CVEs ? If so, this should be mentioned in the changelog and/or forwarded to the security team. New SRUs should not overwrite -security changes - all pockets should have security fixes."

Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello Po-Hsu, or anyone else affected,

Accepted qemu into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:2.5+dfsg-5ubuntu10.25 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in qemu (Ubuntu Xenial):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-xenial
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

# sudo ./run_tests.sh -v
[...]
TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
FAIL emulator
SKIP h_cede_tm (test marked as manual run only)
MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
FAIL sprs

Upgrade to fix (no issues on upgrade)

# sudo ./run_tests.sh -v
[...]
TESTNAME=emulator TIMEOUT=90s ACCEL= ./powerpc/run powerpc/emulator.elf -smp 1
PASS emulator
SKIP h_cede_tm (test marked as manual run only)
MIGRATION=yes TESTNAME=sprs TIMEOUT=90s ACCEL= ./powerpc/run powerpc/sprs.elf -smp 1 -append '-w'
PASS sprs

tags: added: verification-done verification-done-xenial
removed: verification-needed verification-needed-xenial
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package qemu - 1:2.5+dfsg-5ubuntu10.25

---------------
qemu (1:2.5+dfsg-5ubuntu10.25) xenial; urgency=medium

  [ Murilo Opsfelder Araújo ]
  * debian/patches/ubuntu/lp-1723914-ppc64-set-msr-register.patch:
    properly set MSR register for ppc64 targets (LP: #1723914)

  [ Yasmin Beatriz Alves da Silva ]
  * debian/patches/ubuntu/lp-1723904-sprs-backport.patch:
    Fix handling of Power8 special purpose registers (LP: #1723904)

 -- Christian Ehrhardt <email address hidden> Wed, 04 Apr 2018 12:46:15 +0200

Changed in qemu (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for qemu has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Po-Hsu Lin (cypressyew)
Changed in linux (Ubuntu):
status: Confirmed → Invalid
Changed in linux (Ubuntu Xenial):
status: New → Invalid
Changed in linux (Ubuntu Artful):
status: New → Invalid
Frank Heimes (fheimes)
Changed in ubuntu-power-systems:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.