multithreading not working right under qemu user mode for sh4

[Bruno Haible]

In a multithreaded program running under qemu-sh4 (version 2.9.0), thread termination and/or pthread_join is not working right.

The attached program works natively on all kinds of platforms, and under qemu user mode emulation for at least alpha, armelhf, aarch64, powerpc64le.

How to reproduce:
- Compile the program: sh4-linux-gnu-gcc-5 -O -Wall -lpthread -o test-tls test-tls.c
- Set environment variables for running qemu-sh4.
- ~/inst-qemu/2.9.0/bin/qemu-sh4 test-tls

Expected behaviour: After the "Worker xxxxx dying" line, the main() function prints "OK", and the program terminates.

Actual behaviour (only on sh4): After the "Worker xxxxx dying" line, it hangs. Attaching gdb to qemu shows 15 threads with a stack trace like
#0 safe_syscall_base () at /build/qemu-2.9.0/linux-user/host/x86_64/safe-syscall.inc.S:75
#1 0x00005584f86f4c48 in safe_futex (uaddr=<optimized out>, op=op@entry=128, val=val@entry=2, timeout=<optimized out>, uaddr2=uaddr2@entry=0x0,
    val3=val3@entry=-161181992) at /build/qemu-2.9.0/linux-user/syscall.c:921
#2 0x00005584f870353b in do_futex (val3=-161181992, uaddr2=4134624624, timeout=0, val=<optimized out>, op=<optimized out>, uaddr=<optimized out>)
    at /build/qemu-2.9.0/linux-user/syscall.c:7147
#3 do_syscall (cpu_env=<optimized out>, num=240, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=0, arg5=-160342672,
    arg6=-161181992, arg7=0, arg8=0) at /build/qemu-2.9.0/linux-user/syscall.c:11692
#4 0x00005584f86f454a in cpu_loop (env=env@entry=0x5584fb3d04f8) at /build/qemu-2.9.0/linux-user/main.c:2676
#5 0x00005584f86f5dd5 in clone_func (arg=0x7fff4d485c20) at /build/qemu-2.9.0/linux-user/syscall.c:6234
#6 0x00007f08f05a46ba in start_thread (arg=0x7f08f1368700) at pthread_create.c:333
#7 0x00007f08f02da3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

and 1 thread with a stack trace like
#0 safe_syscall_base () at /build/qemu-2.9.0/linux-user/host/x86_64/safe-syscall.inc.S:75
#1 0x00005584f86f4c48 in safe_futex (uaddr=<optimized out>, op=op@entry=0, val=val@entry=18875, timeout=<optimized out>, uaddr2=uaddr2@entry=0x0,
    val3=val3@entry=-161180376) at /build/qemu-2.9.0/linux-user/syscall.c:921
#2 0x00005584f870353b in do_futex (val3=-161180376, uaddr2=4135101768, timeout=0, val=<optimized out>, op=<optimized out>, uaddr=<optimized out>)
    at /build/qemu-2.9.0/linux-user/syscall.c:7147
#3 do_syscall (cpu_env=<optimized out>, num=240, arg1=<optimized out>, arg2=<optimized out>, arg3=<optimized out>, arg4=0, arg5=-159865528,
    arg6=-161180376, arg7=0, arg8=0) at /build/qemu-2.9.0/linux-user/syscall.c:11692
#4 0x00005584f86f454a in cpu_loop (env=0x5584fb3b99a8) at /build/qemu-2.9.0/linux-user/main.c:2676
#5 0x00005584f86c12d3 in main (argc=<optimized out>, argv=0x7fff4d4878b8, envp=<optimized out>)
    at /build/qemu-2.9.0/linux-user/main.c:4860

and 1 thread with a stack trace like
#0 syscall () at ../sysdeps/unix/sysv/linux/x86_64/syscall.S:38
#1 0x00005584f876eab5 in qemu_futex_wait (val=<optimized out>, f=<optimized out>) at /build/qemu-2.9.0/include/qemu/futex.h:26
#2 qemu_event_wait (ev=ev@entry=0x5584faa43d84 <rcu_call_ready_event>) at /build/qemu-2.9.0/util/qemu-thread-posix.c:399
#3 0x00005584f87748ce in call_rcu_thread (opaque=<optimized out>) at /build/qemu-2.9.0/util/rcu.c:249
#4 0x00007f08f05a46ba in start_thread (arg=0x7f08eff62700) at pthread_create.c:333
#5 0x00007f08f02da3dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[Bruno Haible (bruno-clisp)]

Another gnulib test (test-lock) fails with an assertion inside glibc:
test-lock: pthread_mutex_lock.c:81: __pthread_mutex_lock: Assertion `mutex->__data.__owner == 0' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped

Based on this, I would speculate that the problem is in the qemu emulation of the Linux system calls that glibc uses for multithreading or in the implementation of the primitives used by qemu_event_wait().

[Richard Henderson (rth)]

On 07/02/2017 12:53 PM, Bruno Haible wrote:
> In a multithreaded program running under qemu-sh4 (version 2.9.0),
> thread termination and/or pthread_join is not working right.

QEMU does not support the roll-back atomic sequences used by linux on most
uniprocessor SH.

Nor do we support the ll/sc form used by SH4A, although fixing that should be
trivial.

r~

[Thomas Huth (th-huth)]

A patch for this has apparently been included in QEMU v2.10:
https://git.qemu.org/?p=qemu.git;a=commitdiff;h=4bfa602bc2227f5b5a506a4
Is this issue now completely fixed (so that we can close this ticket), or is there something left to do?

[Bruno Haible (bruno-clisp)]

This works fine in qemu-2.10:
$ ~/inst-qemu/2.10.0/bin/qemu-sh4 test-tls
...
Worker 0xecdff4c0 doing value swapping
Worker 0xecdff4c0 doing value swapping
Worker 0xecdff4c0 before final verify
Worker 0xecdff4c0 after final verify
Worker 0xecdff4c0 dying.
 OK