sched_getcpu on ppc64le produces an illegal instruction

[Colin Ian King]

I'm getting SIGILL on a call to sched_getcpu:

gcc 6.3.0 201700221 (Ubuntu 6.3.0-8ubuntu1) and libc6:ppc64el 2.24-7ubuntu2

$ cat example.c
#define _GNU_SOURCE

#include <sched.h>

int main(void)
{
        return sched_getcpu();
}

gcc -g example.c -o example
./example
Illegal instruction

$ gdb ./example
GNU gdb (Ubuntu 7.12.50.20170207-0ubuntu2) 7.12.50.20170207-git
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "powerpc64le-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./example...done.
(gdb) run
Starting program: /home/king/example

Program received signal SIGILL, Illegal instruction.
__kernel_getcpu ()
    at /build/linux-mMU5bT/linux-4.10.0/arch/powerpc/kernel/vdso64/getcpu.S:32
32 /build/linux-mMU5bT/linux-4.10.0/arch/powerpc/kernel/vdso64/getcpu.S: No such file or directory.
(gdb) where
#0 __kernel_getcpu ()
    at /build/linux-mMU5bT/linux-4.10.0/arch/powerpc/kernel/vdso64/getcpu.S:32
#1 0x00003fffb7ea08e8 in sched_getcpu ()
    at ../sysdeps/unix/sysv/linux/sched_getcpu.c:32
#2 0x00000000200007f0 in main () at example.c:7

[Anton Blanchard (anton-samba)]

Hi Colin, what system is this on?

The failing instruction should be a read of SPRG3:

   │0x3fffb7f807c8 <__kernel_getcpu> mfspr r5,259

which should be fine to execute in userspace. There was a NUMA_BALANCING bug around that time where we'd end up mapping the vdso no execute, but I don't think Ubuntu is enabling that feature.

[Adam Conrad (adconrad)]

At least in zesty:

ppc64el/config.common.ppc64el:CONFIG_NUMA_BALANCING=y
ppc64el/config.common.ppc64el:CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y

So, is this glibc bug actually a kernel bug I should assign back?

[Colin Ian King (colin-king)]

Yep, assign back, I'll look at it.

[Colin Ian King (colin-king)]

@Anton:

$ cat /proc/cpuinfo
processor : 0
cpu : POWER8 (raw), altivec supported
clock : 1000.000000MHz
revision : 2.0 (pvr 004d 0200)

timebase : 512000000
platform : pSeries
model : IBM pSeries (emulated by qemu)
machine : CHRP IBM pSeries (emulated by qemu)

[Colin Ian King (colin-king)]

This issue also appears in Ubuntu...
  Xenial 4.4.0-66-generic + libc 2.23-0ubuntu5
  Yakkety 4.8.0-41-generic + libc 2.24-3ubuntu2
  Zesty 4.10.0-11-generic + libc 2.27-7ubuntu2

I've not gone back any further, but it is not a regression in Zesty per se.

And also occurs Debian Jessie 8.8
  3.16.0-4-powerpc64le + libc 2.19-18+deb8u7

I'm using qemu-system-ppc 2.8+dfsg-3ubuntu2

[Anton Blanchard (anton-samba)]

Thanks Colin, I can confirm that this reproduces in upstream QEMU. It looks like none of the user readable SPRGs are implemented. I'm surprised we haven't hit this before!

[Colin Ian King (colin-king)]

I only found this bug with stress-ng; this is a tool that exercises all the system calls.

[Christian Ehrhardt  (paelzer)]

After testing on real hardware and full-virt based on kvm_hv and a clarification with Colin I wanted to make it more clear in the bug that this affects emulation via qemu-system-ppc64le, but not real virtualization.

To some extend that might also explain "I'm surprised we haven't hit this before!" as if it would have been in full virt as well that would have totally puzzled me.