QEMU linux-user emulation is broken for MIPS and POWER in Ubuntu 22.04

[Sergey Evlashev]

[ Impact ]

 * If not used for virtualization qemu is helping by emulating
   other architectures. Here two reproducible issues got
   identified that break such emulation of ppc and mips which
   gladly have been fixed upstream and we could identify the
   changes for. Especially given that 22.04 still has 9 more
   years in the field it seems worth to stabilize/fix this.

 * The patches are rather small and isolated to the
    particular target emulation, they are cleanly applying
    and seem to work fine through tests.

[ Test Plan ]

a. the PPC emulation case

1. Unpack archive [1] and change to test_p64 directory
2. Build the source file with: powerpc64le-linux-gnu-g++ -O2 -static test.cpp -o test_p64
3. Run with QEMU: qemu-ppc64le -cpu POWER9 test_p64 > output.txt
4. Check the output text file output.txt to see the printouts
   With the bug present min/max will identify the wrong number

[1]: https://gitlab.com/qemu-project/qemu/uploads/0e9dbc22e6841496efc15775e6aa624a/test_p64.tar.gz

b. the MIPS emulation case

1. Download the source code from https://github.com/VectorChief/UniSIMD-assembler (master or v1.1.0c)
2. Change to project's test directory and build the binary for MIPS using cross-compiler (see simd_make_m64.mk)
3. Run the binary with QEMU linux-user mode: qemu-mips64el -cpu I6400 simd_test.m64f32Lr6 -c 1 | tee qemu64
4. Check the output text file qemu64 (with pluma or any other text editor) to see the error printouts

[ Where problems could occur ]

 * The changes applied are tied to emulation and that of ppc and
   mips in particular. Therefore these two areas are where we
   need to watch out for unwanted effects. The many other uses
   of qemu should (tm) have on direct chance to be regressed by
   this.

[ Other Info ]

 * This could be two SRU bugs, but after all the context
   is here it would feel weird to split it. But I have
   added two test descriptions to cover each case
   individually.

---

There are issues with QEMU linux-user mode emulation
on MIPS arch:
https://gitlab.com/qemu-project/qemu/-/issues/1531
and on POWER:
https://gitlab.com/qemu-project/qemu/-/issues/1547

The bugs were filed against QEMU version 6.2.0,
however QEMU developers are not normally making point maintenance releases for older versions.

Both bugs for MIPS and POWER architectures were fixed in 7.1.0 and 7.0.0 respectively.
Requesting QEMU 6.2.1 with bug-fixes for Ubuntu 22.04 LTS update cycle.

[Lena Voytek (lvoytek)]

Thank you for the bug report. Based on your tests provided in the upstream bugs I can confirm both are issues in 22.04. Do you happen to know what bug reports upstream are related to these issues being fixed in 7.1.0 and 7.0.0? Also it may be worth submitting a second bug report here for either MIPS or POWER since they are somewhat different issues.

[Christian Ehrhardt  (paelzer)]

I think I was able to identify the patches needed for the mips case and commented in https://gitlab.com/qemu-project/qemu/-/issues/1531 about it asking if there are more related changes.

My assumption is:
857816a42b8 target/mips: Fix store adress of high 64bit in helper_msa_st_b()
ead0bf0d334 target/mips: Do not treat msa INSERT as NOP when wd is zero
4b532b4f2be target/mips: Fix msa checking condition in trans_msa_elm_fn()

Please check the upstream case in a day or two if Philippe knew more about those.

Sadly I'm not 100% sure which of those.

[Christian Ehrhardt  (paelzer)]

For https://gitlab.com/qemu-project/qemu/-/issues/1547 the case is well testable and visible.
If you couldn't spit it, min/max goes nuts in those tests, like:
 farr[11] = 4.318749e+04, farr[3] = 1.468766e+00
-C MIN(farr[11],farr[3]) = 1.468766e+00, MAX(farr[11],farr[3]) = 4.318749e+04
+C MIN(farr[11],farr[3]) = 4.318749e+04, MAX(farr[11],farr[3]) = 4.318749e+04

While executing code like this:
 #define RT_MIN(a, b) ((a) < (b) ? (a) : (b))
 #define RT_MAX(a, b) ((a) > (b) ? (a) : (b))

Optimization makes it use some feature which is broken in emulation.

I see the issue happening with -O1 and higher on the case that was described.
Looking at the assembly I see that the bad code used xsmincdp and xsmaxcdp.

The fix that should be in:
  201fc774e0e target/ppc: Fix xs{max, min}[cj]dp to use VSX registers

I have again asked on the upstream case if anyone knows better.
Rechecking there in a few days would be great.

[Christian Ehrhardt  (paelzer)]

In a hurry I have created a PPA with those changes I identified and thrown a build there
=> https://launchpad.net/~paelzer/+archive/ubuntu/lp-2011832-emulation-issues

I have a WIP branch which is the content that is in that PPA
=> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+ref/lp-2011832-emulation-issues

This isn't perfect, i have not taken much time or care (not even build-tested).
But the upstream patches I found seemed to match the descriptions, where easy to apply and so I wanted to provide something that Sergey can try.

Once confirmed that it works someone might pick up my work and prepare a proper SRU for it?

[Sergey Evlashev (bugreporter66)]

Hi,

I tested the ppa version of QEMU:
sudo add-apt-repository ppa:paelzer/lp-2011832-emulation-issues
sudo apt update

ubuntu-mate@ubuntu-mate:~$
ubuntu-mate@ubuntu-mate:~$ qemu-ppc64le --version
qemu-ppc64le version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa1)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~$

I can confirm that power9 tests are now fixed, but MIPS seems to be failing still.

[Christian Ehrhardt  (paelzer)]

Thanks Sergey,
sadly the mips tests are the harder ones to recreate (at least in the time I had here).
We tried those which seemed to be the ones by their commit message, and that didn't work.

I wonder if you think you could - as fallback - just consider bisecting v7.0.0..v7.1.0 (since you say the latter works)?

Having a second look at the patches, more candidates that might be likely:
 954d1658bde target/mips: Fix SAT_S trans helper
 7fc235c67f6 target/mips: Fix df_extract_val() and df_extract_df() dfe lookup
 1d29f899e7f target/mips: Fix FTRUNC_S and FTRUNC_U trans helper

While I'll update the PPA including those, we get a bit out of the SRU policy and apply patches randomly. Without proper tests we can't be sure what else they might change. Gladly they are all small and reasonable, but we might later on need to filter those to be just those for the case we can test here.

In a bit (if it builds fine) you'll find 6.2+dfsg-2ubuntu6.7~jammyppa3 in the PPA for a re-test.

[Sergey Evlashev (bugreporter66)]

I did the bisecting after cloning the entire QEMU repository and the two commits that fix everything for me are:

7fc235c67f6 target/mips: Fix df_extract_val() and df_extract_df() dfe lookup
1d29f899e7f target/mips: Fix FTRUNC_S and FTRUNC_U trans helper

[Sergey Evlashev (bugreporter66)]

Tested the fixes on top of stock QEMU 6.2.0:

ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix1.diff
patching file target/mips/tcg/msa_translate.c
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix2.diff
patching file target/mips/tcg/msa_translate.c
Hunk #1 succeeded at 747 (offset -5 lines).
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$ patch -p1 < ../../Desktop/qemu_fix3.diff
patching file target/ppc/fpu_helper.c
Hunk #1 succeeded at 2420 (offset -75 lines).
Hunk #2 succeeded at 2455 (offset -75 lines).
patching file target/ppc/helper.h
Hunk #1 succeeded at 392 (offset -11 lines).
patching file target/ppc/translate/vsx-impl.c.inc
ubuntu-mate@ubuntu-mate:~/Downloads/qemu-6.2.0$

All tests pass now for MIPS (qemu_fix1.diff, qemu_fix2.diff) and POWER (qemu_fix3.diff)

[Christian Ehrhardt  (paelzer)]

Thanks,
in the PPA 6.2+dfsg-2ubuntu6.7~jammyppa4 is a respin with just the needed fix for ppc and the two needed fixes for mips.

@Sergey
Please give it a try once built and let me know if it works for both issues.

If that works, here the MR for Jammy so that someone else of the team can have a look if I have mistakes in this rather fast pass.
=> https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439482

[Sergey Evlashev (bugreporter66)]

Tested new build, all tests pass now as expected:

...
========================================================
fully successful test pass writes 212532 bytes to qemu64
the result doesn't depend on CPU type (unlike test64/86)
check the output if qemu64 size differs, check printouts
========================================================
the actual file size after the test run is listed below:
-rw-rw-r-- 1 ubuntu-mate ubuntu-mate 212532 Mar 23 20:39 qemu64
========================================================
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$ qemu-mips64el --version
qemu-mips64el version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa4)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$ qemu-ppc64le --version
qemu-ppc64le version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.7~jammyppa4)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
ubuntu-mate@ubuntu-mate:~/Downloads/UniSIMD-assembler-master/test$

[Christian Ehrhardt  (paelzer)]

A set of security fixes and bug 1921664, bug 2009048 and bug 1994002 are still ongoing and in the SRU process. Once they are completed we can start the SRU of this fix here.

Thanks for your quick replies and testing Sergey!

[Christian Ehrhardt  (paelzer)]

added the SRU template based on the gitlab discussion content

[Christian Ehrhardt  (paelzer)]

FYI: I abandoned the old PPAs and MRs in favor of new ones combining the planned uploads.

PPA: https://launchpad.net/~paelzer/+archive/ubuntu/lp-2011832-emulation-issues-lp-1999885-tod-in-pv-sruprep

MRs:
- J: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439682
- K: https://code.launchpad.net/~paelzer/ubuntu/+source/qemu/+git/qemu/+merge/439683