qemu-system-ppc: Floating point instructions do not properly generate the SPE/Embedded Floating-Point Unavailable interrupt

[Matthieu Bucchianeri]

When emulating certain floating point instructions or vector instructions on PowerPC machines, QEMU does not properly generate the SPE/Embedded Floating-Point Unavailable interrupt.

As described in the Signal Processing Engine (SPE) Programming Environments Manual, Rev. 0, available at https://www.nxp.com/docs/en/reference-manual/SPEPEM.pdf:

> An SPE/embedded floating-point unavailable exception occurs on an attempt to execute any of the
> following instructions and MSR[SPV] is not set:
> * SPE instruction (except brinc)
> * An embedded scalar double-precision instruction
> * A vector single-precision floating-point instructions
> It is not used by embedded scalar single-precision floating-point instructions

This behavior was partially reported in Bug #1611394, however the issue is larger than what is described in that bug. As mentioned in that bug, some single-precision instructions generate the exception (while they should not), which is incorrect but does not typically produce an incorrect output. What is more of an issue is that several double-precision and vector instructions do not generate the exception (while they should), and this breaks support for lazy FPU/vector context switching in Linux (for example).

The upper 32-bit of the double-precision/vector registers (which are in fact hidden in the general purpose registers) is not properly saved/restored, and this causes arithmetic errors. This was observed very frequently on a commercial project that does a lot of double-precision computations. The application works perfectly fine on an MPC8548 CPU, but fails often with QEMU.

This is only an issue with full platform emulation - the SPE/Embedded Floating-Point Unavailable interrupt is not relevant for application emulation.

The issue can be reproduced using the attached Linux program "spe-bug.c". This program properly prints the number 42 (as the result of some very simple double-precision computation) on real PowerPC hardware, but prints an incorrect result (typically 0) on QEMU.

This issue was first discovered in an older version of QEMU, but is also reproduced in the latest:

# git rev-parse HEAD
7adfbea8fd1efce36019a0c2f198ca73be9d3f18
# ppc-softmmu/qemu-system-ppc --version
QEMU emulator version 5.0.91 (v5.1.0-rc1-28-g7adfbea8fd-dirty)
Copyright (c) 2003-2020 Fabrice Bellard and the QEMU Project developers

Upon further analysis a total of 39 instructions are misbehaving:

efsabs: raised: 1, expected: 0
efsnabs: raised: 1, expected: 0
efsneg: raised: 1, expected: 0
efdcfs: raised: 0, expected: 1
efdcfsf: raised: 0, expected: 1
efdcfsi: raised: 0, expected: 1
efdcfuf: raised: 0, expected: 1
efdcfui: raised: 0, expected: 1
efdctsf: raised: 0, expected: 1
efdctsi: raised: 0, expected: 1
efdctsiz: raised: 0, expected: 1
efdctuf: raised: 0, expected: 1
efdctui: raised: 0, expected: 1
efdctuiz: raised: 0, expected: 1
efscfd: raised: 0, expected: 1
evfscfsf: raised: 0, expected: 1
evfscfsi: raised: 0, expected: 1
evfscfuf: raised: 0, expected: 1
evfscfui: raised: 0, expected: 1
evfsctsf: raised: 0, expected: 1
evfsctsi: raised: 0, expected: 1
evfsctsiz: raised: 0, expected: 1
evfsctuf: raised: 0, expected: 1
evfsctui: raised: 0, expected: 1
evfsctuiz: raised: 0, expected: 1
brinc: raised: 0, expected: 1
efsadd: raised: 1, expected: 0
efsdiv: raised: 1, expected: 0
efsmul: raised: 1, expected: 0
efssub: raised: 1, expected: 0
evsplatfi: raised: 0, expected: 1
evsplati: raised: 0, expected: 1
efscmpeq: raised: 1, expected: 0
efscmpgt: raised: 1, expected: 0
efscmplt: raised: 1, expected: 0
efststeq: raised: 1, expected: 0
efststgt: raised: 1, expected: 0
efststlt: raised: 1, expected: 0
evsel: raised: 0, expected: 1

When "raised" is 0 and "expected" is 1, this means that the SPE/Embedded Floating-Point Unavailable interrupt was not generated while it should have.
When "raised" is 1 and "expected" is 0, this means that the SPE/Embedded Floating-Point Unavailable interrupt was generated while it should not have (Bug #1611394).

A comprehensive program testing all the instructions listed in the Signal Processing Engine (SPE) Programming Environments Manual, Rev. 0 is posted in the comments of this ticket, and can be used to reproduce the issue, and validate the future fix.

[Matthieu Bucchianeri (matthieu-bucchianeri)]

Attaching the test program mentioned in the description. This program (a kernel module, in fact) can be loaded on a Linux system both in QEMU or on real hardware.

In the next comments, I will attach the detailed output of the test program.

[Matthieu Bucchianeri (matthieu-bucchianeri)]

Attaching the output of the test module (above) when run on a real MPC8548. This is to establish a baseline validating which instructions shall or shall not generate the SPE/Embedded Floating-Point Unavailable interrupt.

Note that on the MPC8548, it is observed that the "brinc" instruction does generate the interrupt, which contradicts section 4.2.3 SPE/Embedded Floating-Point Unavailable Interrupt of the Signal Processing Engine (SPE) Programming Environments Manual, Rev. 0 (see the quote in the description). The test program was modified to pass 100% on real hardware, hence claiming that "brinc" shall generate the interrupt.

[Matthieu Bucchianeri (matthieu-bucchianeri)]

Attaching the output of the test module run on QEMU. As shown in the description, 39 instructions do not comply with the rule described in the Signal Processing Engine (SPE) Programming Environments Manual, Rev. 0.

QEMU was run as follows:

# qemu/ppc-softmmu/qemu-system-ppc -nographic -machine mpc8544ds -kernel buildroot/output/images/uImage -hda buildroot/output/images/rootfs.cpio -append "root=/dev/sda rw single"

The Linux image is built using buildroot, and the selected configuration is qemu_ppc_mpc8544ds_defconfig with some very mild changes to setup an overlay and initramfs.

Then, the module is loaded, output can be observed directly in the console, or using dmesg:

# insmod spe-test.ko

[Matthieu Bucchianeri (matthieu-bucchianeri)]

I have already written a patch for this issue, that I will be submitting to the community in the next few days.

[Matthieu Bucchianeri (matthieu-bucchianeri)]

> Note that on the MPC8548, it is observed that the "brinc"
> instruction does generate the interrupt, which contradicts
> section 4.2.3 SPE/Embedded Floating-Point Unavailable Interrupt
> of the Signal Processing Engine (SPE) Programming Environments
> Manual, Rev. 0 (see the quote in the description). The test
> program was modified to pass 100% on real hardware, hence
> claiming that "brinc" shall generate the interrupt.

I have actually dug up some more on this and changed my mind. There are more references in the PowerPC documentation indicating that "brinc" shall NOT generate the interrupt.

In the EREF: A Programmer’s Reference Manual for Freescale Power Architecture Processors, Rev. 1 (EIS 2.1), Table 4-22. MSR Field Descriptions clearly states that when SPV==0:

> The processor cannot execute any category SPE, SP.FD, or SP.FV
> instructions except for the brinc instruction.

The patch that I am submitting to fix this bug will leave the behavior of "brinc" unchanged (ie: to not generate the interrupt).

[Matthieu Bucchianeri (matthieu-bucchianeri)]

Testing performed on the patch sent to the mailing list:

1) make check

2) Run the "spe-bug.c" userspace program, observed the correct result:

# qemu/ppc-softmmu/qemu-system-ppc -nographic -machine mpc8544ds -kernel buildroot/output/images/uImage -hda buildroot/output/images/rootfs.cpio -append "root=/dev/sda rw single"
[...]
Run /init as init process
# ./spe-bug
42.000000
#

This used to return 0.000000 without the fix.

3) Run the "spe-test" test module, and observed the correct result:

# qemu/ppc-softmmu/qemu-system-ppc -nographic -machine mpc8544ds -kernel buildroot/output/images/uImage -hda buildroot/output/images/rootfs.cpio -append "root=/dev/sda rw single"
[...]
Run /init as init process
# insmod spe-test.ko
spe_test: loading out-of-tree module taints kernel.
Begin testing...
Testing efdabs
SPE used in kernel (task=(ptrval), pc=c9042048)
[...]
No errors.
End testing.

This used to count 39 errors. Attached the full output as well.

Note that per my previous comment, I have modified the "spe-test" code to expect that the "brinc" instruction does not generate the interrupt, using this diff from the previously attached "spe-test" source code:

--- spe-test.c.orig 2020-07-25 12:22:23.628315553 -0700
+++ spe-test.c 2020-07-24 23:12:27.508234566 -0700
@@ -84,7 +84,7 @@
     TEST_INSN_RD_RA (evfsctuf, 1) \
     TEST_INSN_RD_RA (evfsctui, 1) \
     TEST_INSN_RD_RA (evfsctuiz, 1) \
- TEST_INSN_RD_RA_RB (brinc, 1) \
+ TEST_INSN_RD_RA_RB (brinc, 0) \
     TEST_INSN_RD_RA_RB (efdadd, 1) \
     TEST_INSN_RD_RA_RB (efddiv, 1) \
     TEST_INSN_RD_RA_RB (efdmul, 1) \

[John Snow (jnsnow)]

Assuming that this commit:
https://gitlab.com/qemu-project/qemu/-/commit/8dcdb535d7cc4ba6270bb756e12e1d323254ed4e

is sufficient to mark this bug as Fix Committed. Please re-open if I am mistaken.

[Thomas Huth (th-huth)]

Released with QEMU v5.2.0.