QEMU linux-user emulation is broken for MIPS and POWER in Ubuntu 22.04

[Sergey Evlashev]

[ Impact ]

 * If not used for virtualization qemu is helping by emulating
   other architectures. Here two reproducible issues got
   identified that break such emulation of ppc and mips which
   gladly have been fixed upstream and we could identify the
   changes for. Especially given that 22.04 still has 9 more
   years in the field it seems worth to stabilize/fix this.

 * The patches are rather small and isolated to the
    particular target emulation, they are cleanly applying
    and seem to work fine through tests.

[ Test Plan ]

a. the PPC emulation case

1. Unpack archive [1] and change to test_p64 directory
2. Build the source file with: powerpc64le-linux-gnu-g++ -O2 -static test.cpp -o test_p64
3. Run with QEMU: qemu-ppc64le -cpu POWER9 test_p64 > output.txt
4. Check the output text file output.txt to see the printouts
   With the bug present min/max will identify the wrong number

[1]: https://gitlab.com/qemu-project/qemu/uploads/0e9dbc22e6841496efc15775e6aa624a/test_p64.tar.gz

b. the MIPS emulation case

1. Download the source code from https://github.com/VectorChief/UniSIMD-assembler (master or v1.1.0c)
2. Change to project's test directory and build the binary for MIPS using cross-compiler (see simd_make_m64.mk)
3. Run the binary with QEMU linux-user mode: qemu-mips64el -cpu I6400 simd_test.m64f32Lr6 -c 1 | tee qemu64
4. Check the output text file qemu64 (with pluma or any other text editor) to see the error printouts

[ Where problems could occur ]

 * The changes applied are tied to emulation and that of ppc and
   mips in particular. Therefore these two areas are where we
   need to watch out for unwanted effects. The many other uses
   of qemu should (tm) have on direct chance to be regressed by
   this.

[ Other Info ]

 * This could be two SRU bugs, but after all the context
   is here it would feel weird to split it. But I have
   added two test descriptions to cover each case
   individually.

---

There are issues with QEMU linux-user mode emulation
on MIPS arch:
https://gitlab.com/qemu-project/qemu/-/issues/1531
and on POWER:
https://gitlab.com/qemu-project/qemu/-/issues/1547

The bugs were filed against QEMU version 6.2.0,
however QEMU developers are not normally making point maintenance releases for older versions.

Both bugs for MIPS and POWER architectures were fixed in 7.1.0 and 7.0.0 respectively.
Requesting QEMU 6.2.1 with bug-fixes for Ubuntu 22.04 LTS update cycle.

[Lena Voytek (lvoytek)]

Thank you for the bug report. Based on your tests provided in the upstream bugs I can confirm both are issues in 22.04. Do you happen to know what bug reports upstream are related to these issues being fixed in 7.1.0 and 7.0.0? Also it may be worth submitting a second bug report here for either MIPS or POWER since they are somewhat different issues.

[Christian Ehrhardt  (paelzer)]

I think I was able to identify the patches needed for the mips case and commented in https://gitlab.com/qemu-project/qemu/-/issues/1531 about it asking if there are more related changes.

My assumption is:
857816a42b8 target/mips: Fix store adress of high 64bit in helper_msa_st_b()
ead0bf0d334 target/mips: Do not treat msa INSERT as NOP when wd is zero
4b532b4f2be target/mips: Fix msa checking condition in trans_msa_elm_fn()

Please check the upstream case in a day or two if Philippe knew more about those.

Sadly I'm not 100% sure which of those.

[Christian Ehrhardt  (paelzer)]

For https://gitlab.com/qemu-project/qemu/-/issues/1547 the case is well testable and visible.
If you couldn't spit it, min/max goes nuts in those tests, like:
 farr[11] = 4.318749e+04, farr[3] = 1.468766e+00
-C MIN(farr[11],farr[3]) = 1.468766e+00, MAX(farr[11],farr[3]) = 4.318749e+04
+C MIN(farr[11],farr[3]) = 4.318749e+04, MAX(farr[11],farr[3]) = 4.318749e+04

While executing code like this:
 #define RT_MIN(a, b) ((a) < (b) ? (a) : (b))
 #define RT_MAX(a, b) ((a) > (b) ? (a) : (b))

Optimization makes it use some feature which is broken in emulation.

I see the issue happening with -O1 and higher on the case that was described.
Looking at the assembly I see that the bad code used xsmincdp and xsmaxcdp.

The fix that should be in:
  201fc774e0e target/ppc: Fix xs{max, min}[cj]dp to use VSX registers

I have again asked on the upstream case if anyone knows better.
Rechecking there in a few days would be great.

[Christian Ehrhardt  (paelzer)]

In a hurry I have created a PPA with those changes I identified and thrown a build there
=> https://launchpad.net/~paelzer/+archive/ubuntu/lp-2011832-emulation-issues

I have a WIP branch which is the content that is in that PPA
=> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+ref/lp-2011832-emulation-issues

This isn't perfect, i have not taken much time or care (not even build-tested).
But the upstream patches I found seemed to match the descriptions, where easy to apply and so I wanted to provide something that Sergey can try.

Once confirmed that it works someone might pick up my work and prepare a proper SRU for it?

[Sergey Evlashev (bugreporter66)]

Hi,

I tested the ppa version of QEMU:
sudo add-apt-repository ppa:paelzer/lp-2011832-emulation-issues
sudo apt update

ubuntu-mate@ubuntu-mate:~$
ubuntu-mate@ubuntu-mate:~$ qemu-ppc64le --version
qemu-ppc64le version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~$

I can confirm that power9 tests are now fixed, but MIPS seems to be failing still.

[Christian Ehrhardt  (paelzer)]

Thanks Sergey,
sadly the mips tests are the harder ones to recreate (at least in the time I had here).
We tried those which seemed to be the ones by their commit message, and that didn't work.

I wonder if you think you could - as fallback - just consider bisecting v7.0.0..v7.1.0 (since you say the latter works)?

Having a second look at the patches, more candidates that might be likely:
 954d1658bde target/mips: Fix SAT_S trans helper
 7fc235c67f6 target/mips: Fix df_extract_val() and df_extract_df() dfe lookup
 1d29f899e7f target/mips: Fix FTRUNC_S and FTRUNC_U trans helper

While I'll update the PPA including those, we get a bit out of the SRU policy and apply patches randomly. Without proper tests we can't be sure what else they might change. Gladly they are all small and reasonable, but we might later on need to filter those to be just those for the case we can test here.

In a bit (if it builds fine) you'll find 6.2+dfsg-2ubuntu6.7~jammyppa3 in the PPA for a re-test.

[Sergey Evlashev (bugreporter66)]

I did the bisecting after cloning the entire QEMU repository and the two commits that fix everything for me are:

7fc235c67f6 target/mips: Fix df_extract_val() and df_extract_df() dfe lookup
1d29f899e7f target/mips: Fix FTRUNC_S and FTRUNC_U trans helper

[Sergey Evlashev (bugreporter66)]

Tested the fixes on top of stock QEMU 6.2.0:

ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix1.diff
patching file target/mips/tcg/msa_translate.c
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix2.diff
patching file target/mips/tcg/msa_translate.c
Hunk #1 succeeded at 747 (offset -5 lines).
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix3.diff
patching file target/ppc/fpu_helper.c
Hunk #1 succeeded at 2420 (offset -75 lines).
Hunk #2 succeeded at 2455 (offset -75 lines).
patching file target/ppc/helper.h
Hunk #1 succeeded at 392 (offset -11 lines).
patching file target/ppc/translate/vsx-impl.c.inc
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$

All tests pass now for MIPS (qemu_fix1.diff, qemu_fix2.diff) and POWER (qemu_fix3.diff)

[Christian Ehrhardt  (paelzer)]

Thanks,
in the PPA 6.2+dfsg-2ubuntu6.7~jammyppa4 is a respin with just the needed fix for ppc and the two needed fixes for mips.

@Sergey
Please give it a try once built and let me know if it works for both issues.

If that works, here the MR for Jammy so that someone else of the team can have a look if I have mistakes in this rather fast pass.
=> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439482

[Sergey Evlashev (bugreporter66)]

Tested new build, all tests pass now as expected:

...
========================================================
fully successful test pass writes 212532 bytes to qemu64
the result doesn't depend on CPU type (unlike test64/86)
check the output if qemu64 size differs, check printouts
========================================================
the actual file size after the test run is listed below:
-rw-rw-r-- 1 ubuntu-mate ubuntu-mate 212532 Mar 23 20:39 qemu64
========================================================
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$ qemu-mips64el --version
qemu-mips64el version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa4)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$ qemu-ppc64le --version
qemu-ppc64le version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa4)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$

[Christian Ehrhardt  (paelzer)]

A set of security fixes and bug 1921664, bug 2009048 and bug 1994002 are still ongoing and in the SRU process. Once they are completed we can start the SRU of this fix here.

Thanks for your quick replies and testing Sergey!

[Christian Ehrhardt  (paelzer)]

added the SRU template based on the gitlab discussion content

[Christian Ehrhardt  (paelzer)]

FYI: I abandoned the old PPAs and MRs in favor of new ones combining the planned uploads.

PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-2011832-emulation-issues-lp-1999885-tod-in-pv-sruprep

MRs:
- J: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439682
- K: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439683

[Christian Ehrhardt  (paelzer)]

The former SRUs cleared, now uploaded to -unapproved

[Steve Langasek (vorlon)]

Hello Sergey, or anyone else affected,

Accepted qemu into kinetic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:7.0+dfsg-7ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-kinetic to verification-done-kinetic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-kinetic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

Hello Sergey, or anyone else affected,

Accepted qemu into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/qemu/1:6.2+dfsg-2ubuntu6.8 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:7.0+dfsg-7ubuntu2.3) for kinetic have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/2.2+22.04ubuntu3 (arm64, ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/kinetic/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted qemu (1:6.2+dfsg-2ubuntu6.8) for jammy have finished running.
The following regressions have been reported in tests triggered by the package:

initramfs-tools/0.140ubuntu13.1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/jammy/update_excuses.html#qemu

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Christian Ehrhardt  (paelzer)]

FYI: The reported tests were just flaky (not due to qemu) and are resolved now.

[Christian Ehrhardt  (paelzer)]

Checking powerpc on jammy
root@j:~# qemu-ppc64le -cpu POWER9 test_p64.O1 > output.O1.txt
root@j:~# qemu-ppc64le -cpu POWER9 test_p64.O0 > output.O0.txt
root@j:~# qemu-ppc64le -cpu POWER9 test_p64.O2 > output.O2.txt
root@j:~# md5sum output.O0.txt output.O1.txt output.O2.txt
b569dea800dcc2ec7312cc3455a124e0 output.O0.txt
b569dea800dcc2ec7312cc3455a124e0 output.O1.txt
b569dea800dcc2ec7312cc3455a124e0 output.O2.txt

They now all behave the same with
root@j:~# apt-cache policy qemu-system-x86
qemu-system-x86:
  Installed: 1:6.2+dfsg-2ubuntu6.8
  Candidate: 1:6.2+dfsg-2ubuntu6.8
  Version table:
 *** 1:6.2+dfsg-2ubuntu6.8 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

The same case on the same system failed before creating wrong results.

[Christian Ehrhardt  (paelzer)]

Build test workload:
$ git clone https://github.com/VectorChief/UniSIMD-assembler.git
$ cd UniSIMD-assembler/test
$ sudo apt install g++-mipsisa64r6el-linux-gnuabi64 g++-mipsisa64r6-linux-gnuabi64
# (replace mips-mti-linux-gnu with mipsisa64r6el-linux-gnuabi64 for LE)
# (replace mips-mti-linux-gnu with mipsisa64r6-linux-gnuabi64 for BE)
$ make -f simd_make_m64.mk

$ apt install qemu-user
$ qemu-mips64el -cpu I6400 simd_test.m64f32Lr6 -c 1 | tee qemu64

Versions compared:

Kinetic:
 Version table:
 *** 1:7.0+dfsg-7ubuntu2.3 500
        500 http://archive.ubuntu.com/ubuntu kinetic-proposed/universe amd64 Packages
        100 /var/lib/dpkg/status
     1:7.0+dfsg-7ubuntu2.2 500
        500 http://archive.ubuntu.com/ubuntu kinetic-updates/universe amd64 Packages
     1:7.0+dfsg-7ubuntu2.1 500

Jammy:
  Version table:
     1:6.2+dfsg-2ubuntu6.8 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 Packages
 *** 1:6.2+dfsg-2ubuntu6.7 500
        500 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages
        100 /var/lib/dpkg/status

Indeed without the fix both releases spew out errors in the self check like

# diff -Naur qemu64.old qemu64.new
--- qemu64.old 2023-04-18 12:20:26.176053632 +0000
+++ qemu64.new 2023-04-18 12:17:35.263217743 +0000
@@ -9,8 +9,8 @@
 --------------------------------------------------------
 Test-redundant overridden: 1
 -------------------- SUB TEST = 1 - ptr/fp = 64f32 --
-Time C = 1
-Time S = 0
+Time C = 0
+Time S = 1
 -------------------------------------- simd = 128x2v1 -
 -------------------- SUB TEST = 2 - ptr/fp = 64f32 --
 Time C = 0
@@ -18,7 +18,7 @@
 -------------------------------------- simd = 128x2v1 -
 -------------------- SUB TEST = 3 - ptr/fp = 64f32 --
 Time C = 0
-Time S = 1
+Time S = 0
 -------------------------------------- simd = 128x2v1 -
 -------------------- SUB TEST = 4 - ptr/fp = 64f32 --
 Time C = 0
@@ -37,486 +37,72 @@
 Time S = 0
 -------------------------------------- simd = 128x2v1 -
 -------------------- SUB TEST = 8 - ptr/fp = 64f32 --
-Time C = 1
+Time C = 0
 Time S = 0
-iarr[23] = 318773
-C iarr[23]+((rt_uelm)+iarr[23]<<1) = 2666648, iarr[23]-((rt_uelm)-iarr[23]>>2) = 637546
-S iarr[23]+((rt_uelm)+iarr[23]<<1) = 637546, iarr[23]-((rt_uelm)-iarr[23]>>2) = 637546
-iarr[22] = 7665
-C iarr[22]+((rt_uelm)+iarr[22]<<1) = 87655196, iarr[22]-((rt_uelm)-iarr[22]>>2) = 15330
-S iarr[22]+((rt_uelm)+iarr[22]<<1) = 15330, iarr[22]-((rt_uelm)-iarr[22]>>2) = 15330
-iarr[21] = 87647531

Those are from the self-tests and detect issues.
With the new version they are gone and fixed.
With ppc and mips issues verified I'm marking the bug as that.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:6.2+dfsg-2ubuntu6.8

---------------
qemu (1:6.2+dfsg-2ubuntu6.8) jammy; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips and powerpc (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for qemu has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package qemu - 1:7.0+dfsg-7ubuntu2.3

---------------
qemu (1:7.0+dfsg-7ubuntu2.3) kinetic; urgency=medium

  * d/p/u/lp-1999885-s390x-tod-kvm-don-t-save-restore-the-TOD-in-PV-guest.patch:
    avoid timer issues in s390x secure execution guests (LP: #1999885)
  * d/p/u/lp-2011832-*: fix emulation issues in mips (LP: #2011832)

 -- Christian Ehrhardt <email address hidden> Thu, 23 Mar 2023 08:18:28 +0100