ELF loader fails to load shared object on ThunderX2 running RHEL7

[Caroline Concatto]

Simple test:
hello.c

include <stdio.h>

int main(int argc, char* argv[])
{
  {
    printf("Hello World... \n");
  }
  return 0;
}

when compiled with :
*Compiler
https://developer.arm.com/tools-and-software/server-and-hpc/arm-architecture-tools/arm-allinea-studio/download
Arm-Compiler-for-HPC_19.3_RHEL_7_aarch64.tar

*Running:
1) with -armpl
     armclang -armpl hello.c
     ./qemu/build/aarch64-linux-user/qemu-aarch64 a.out
2) without flag
    armclang hello.c
     ./qemu/build/aarch64-linux-user/qemu-aarch64 a.out

•With Docker image:
       CentOS Linux release 7.7.1908 (AltArch)

*Two different machines:
       AArch64, Taishan. tsv110, Kunpeng 920, ARMv8.2-A
       AArch64, Taishan 2280, Cortex-A72, ARMv8-A

*QEMU 4.0
     qemu-aarch64 version 4.1.91 (v4.2.0-rc1)

Results:

 ****Taishan 2280 Cortex-A72
      Running
1)with -armpl flag with and without the docker
          WORKS-> Hello World...
               -> ldd a.out
ldd a.out
linux-vdso.so.1 => (0x0000ffffbc6a2000)
libamath_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libamath_generic.so (0x0000ffffbc544000)
libm.so.6 => /lib64/libm.so.6 (0x0000ffffbc493000)
libastring_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libastring_generic.so (0x0000ffffbc472000) libarmflang.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libarmflang.so (0x0000ffffbbfd3000)
libomp.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libomp.so (0x0000ffffbbef5000)
librt.so.1 => /lib64/librt.so.1 (0x0000ffffbbed4000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffffbbe9f000)
libarmpl_lp64_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libarmpl_lp64_generic.so (0x0000ffffb3306000)
libc.so.6 => /lib64/libc.so.6 (0x0000ffffb3180000)
libstdc++.so.6 => /scratch/gcc-9.2.0_Generic-AArch64_RHEL-8_aarch64-linux/lib64/libstdc++.so.6 (0x0000ffffb2f30000)
libgcc_s.so.1 => /scratch/gcc-9.2.0_Generic-AArch64_RHEL-8_aarch64-linux/lib64/libgcc_s.so.1 (0x0000ffffb2eff000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000ffffb2ede000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffbc674000)

Running
2) without -armpl flag with and without the docker
           WORKS -> Hello World...
                 -> ldd a.out
ldd a.out
 linux-vdso.so.1 => (0x0000ffffa6895000)
libastring_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libastring_generic.so (0x0000ffffa6846000)
libc.so.6 => /lib64/libc.so.6 (0x0000ffffa66c0000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffa6867000)

****Taishan - tsv110 Kunpeng 920
       For Running

1)with -armpl flag with and without the docker
           DOES NOT WORK -> with and without Docker
                         -> It shows : qemu:handle_cpu_signal received signal outside vCPU
 context @ pc=0xffffaaa8844a
                         -> ldd a.out
ldd a.out
linux-vdso.so.1 => (0x0000ffffad4b0000)
libamath_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libamath_generic.so (0x0000ffffad370000)
libm.so.6 => /lib64/libm.so.6 (0x0000ffffad2a0000)
libastring_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libastring_generic.so (0x0000ffffad270000) libarmflang.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libarmflang.so (0x0000ffffacdd0000)
libomp.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libomp.so (0x0000ffffaccf0000)
librt.so.1 => /lib64/librt.so.1 (0x0000ffffaccc0000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffffacc80000)
libarmpl_lp64_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libarmpl_lp64_generic.so (0x0000ffffa40e0000)
libc.so.6 => /lib64/libc.so.6 (0x0000ffffa3f50000)
libstdc++.so.6 => /scratch/gcc-9.2.0_Generic-AArch64_RHEL-8_aarch64-linux/lib64/libstdc++.so.6 (0x0000ffffa3d00000)
libgcc_s.so.1 => /scratch/gcc-9.2.0_Generic-AArch64_RHEL-8_aarch64-linux/lib64/libgcc_s.so.1 (0x0000ffffa3cc0000)
libdl.so.2 => /lib64/libdl.so.2 (0x0000ffffa3c90000)
/lib/ld-linux-aarch64.so.1 (0x0000ffffad4c0000)

Running
2) without -armpl flag with and without the docker
               WORKS -> Hello World..
                     -> ldd a.out
ldd a.out
linux-vdso.so.1 => (0x0000ffff880c0000)
libastring_generic.so => /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libastring_generic.so (0x0000ffff88080000)
libc.so.6 => /lib64/libc.so.6 (0x0000ffff87ee0000)
/lib/ld-linux-aarch64.so.1 (0x0000ffff880d0000)

[Alex Bennée (ajbennee)]

Could you invoke one of the failing and passing cases with -d page and post the results please.

[Caroline Concatto (carolineconcatto)]

 ****Taishan 2280 Cortex-A72
      Running
1)with -armpl flag with and without the docker
  armclang -armpl hello.c
 ./qemu/build/aarch64-linux-user/qemu-aarch64 -d page a.out

host mmap_min_addr=0x8000
Reserved 0x21000 bytes of guest address space
Relocating guest address space from 0x0000000000400000 to 0x288a000
guest_base 0x248a000
start end size prot
0000000000400000-0000000000401000 0000000000001000 r-x
000000000041f000-0000000000421000 0000000000002000 rw-
0000004000000000-0000004000001000 0000000000001000 ---
0000004000001000-0000004000801000 0000000000800000 rw-
0000004000801000-000000400081f000 000000000001e000 r-x
000000400081f000-0000004000830000 0000000000011000 ---
0000004000830000-0000004000833000 0000000000003000 rw-
start_brk 0x0000000000000000
end_code 0x00000000004009f4
start_code 0x0000000000400000
start_data 0x000000000041fd68
end_data 0x0000000000420024
start_stack 0x0000004000800510
brk 0x0000000000420028
entry 0x00000040008020e0
argv_start 0x0000004000800518
env_start 0x0000004000800528
auxv_start 0x0000004000800588
Hello World...

****Taishan - tsv110 Kunpeng 920
       For Running

1)with -armpl flag with and without the docker
  armclang -armpl hello.c
 ./qemu/build/aarch64-linux-user/qemu-aarch64 -d page a.out

host mmap_min_addr=0x1000
Reserved 0x30000 bytes of guest address space
Relocating guest address space from 0x0000000000400000 to 0x2890000
guest_base 0x2490000
start end size prot
0000000000400000-0000000000410000 0000000000010000 r-x
0000000000410000-0000000000430000 0000000000020000 rw-
0000004000000000-0000004000010000 0000000000010000 ---
0000004000010000-0000004000810000 0000000000800000 rw-
0000004000810000-0000004000830000 0000000000020000 r-x
0000004000830000-0000004000850000 0000000000020000 rw-
start_brk 0x0000000000000000
end_code 0x00000000004009f4
start_code 0x0000000000400000
start_data 0x000000000041fd68
end_data 0x0000000000420024
start_stack 0x000000400080f560
brk 0x0000000000420028
entry 0x00000040008110e0
argv_start 0x000000400080f568
env_start 0x000000400080f578
auxv_start 0x000000400080f5d8
qemu:handle_cpu_signal received signal outside vCPU context @ pc=0xffffb1938536

[Alex Bennée (ajbennee)]

As it's taking longer to get the compiler up and running on my system could you attach the failing binary along with the extra .so libs from /scratch/arm-linux-compiler/*

[Assad Hashmi (a-hashmi)]

For info, a similar type of failure has been seen when loading libarmflang.so on DynamoRIO:
https://github.com/DynamoRIO/dynamorio/issues/3385
It's to do with the .dynstr section being mapped incorrectly causing a SIGBUS.

[Alex Bennée (ajbennee)]

I've attempted to replicate but it works for me:

16:55:37 [alex@idun:~/l/t/hello-armpl] $ ~/lsrc/qemu.git/builds/all/aarch64-linux-user/qemu-aarch64 ./hello-armpl
Hello World...
16:55:52 [alex@idun:~/l/t/hello-armpl] $ ldd ./hello-armpl
        linux-vdso.so.1 (0x0000ffffb9e78000)
        libamath_generic.so => /home/alex/lsrc/tests/hello-armpl/libamath_generic.so (0x0000ffffb9d1a000)
        libm.so.6 => /lib64/libm.so.6 (0x0000ffffb9c50000)
        libastring_generic.so => /home/alex/lsrc/tests/hello-armpl/libastring_generic.so (0x0000ffffb9c2f000)
        libarmflang.so => /home/alex/lsrc/tests/hello-armpl/libarmflang.so (0x0000ffffb97b2000)
        libomp.so => /home/alex/lsrc/tests/hello-armpl/libomp.so (0x0000ffffb96d4000)
        librt.so.1 => /lib64/librt.so.1 (0x0000ffffb96bc000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffffb968a000)
        libarmpl_lp64_generic.so => /home/alex/lsrc/tests/hello-armpl/libarmpl_lp64_generic.so (0x0000ffffb0e12000)
        libc.so.6 => /lib64/libc.so.6 (0x0000ffffb0c95000)
        /lib/ld-linux-aarch64.so.1 => /lib64/ld-linux-aarch64.so.1 (0x0000ffffb9e4a000)
        libstdc++.so.6 => /home/alex/lsrc/tests/hello-armpl/libstdc++.so.6 (0x0000ffffb0a9c000)
        libgcc_s.so.1 => /home/alex/lsrc/tests/hello-armpl/libgcc_s.so.1 (0x0000ffffb0a6b000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000ffffb0a57000)

[Caroline Concatto (carolineconcatto)]

Hi Alex,

So, it works in some machines and others not. Mainly in machines with RHEL OS that we found the problem.
What is the OS you are using?

[Alex Bennée (ajbennee)]

This was on Aarch64 Ubuntu 18.04 - I don't have any RHEL machines around but if you send the ld.so along with the other libraries that won't matter in replicating the fault on my x86 host.

[Peter Maydell (pmaydell)]

IIRC RHEL uses 64k pages but Ubuntu does not -- maybe that is relevant ? Is the guest binary built for 4K or 64K pages?

[Caroline Concatto (carolineconcatto)]

Alex,

Do you have the licence to run the compiler library?

[Alex Bennée (ajbennee)]

I do have a ARM HPC compiler license which I assume includes the armpl blobs that came with it. You can email me directly at my Linaro email (<email address hidden>) if you don't want to upload the test case here.

[Alex Bennée (ajbennee)]

FWIW -p 65536 doesn't trigger anything although I wouldn't trust -p too much:

env LD_LIBRARY_PATH=/opt/arm/armpl-19.3.0_ThunderX2CN99_Ubuntu-16.04_arm-hpc-compiler_19.3_aarch64-linux/lib/:/opt/arm/arm-hpc-compiler-19.3_Generic-AArch64_Ubuntu-16.04_aarch64-linux/lib/ ~/lsrc/qemu.git/aarch64-linux-user/qemu-aarch64 -p 65536 -d page ./hello-armpl
host mmap_min_addr=0x8000
Reserved 0x20000 bytes of guest address space
Relocating guest address space from 0x0000000000400000 to 0x400000
guest_base 0x0
start end size prot
0000000000400000-0000000000403000 0000000000003000 r-x
0000000000410000-0000000000413000 0000000000003000 rw-
0000004000000000-0000004000001000 0000000000001000 ---
0000004000001000-0000004000801000 0000000000800000 rw-
0000004000810000-000000400082f000 000000000001f000 r-x
000000400082f000-0000004000830000 0000000000001000 ---
0000004000830000-000000400083f000 000000000000f000 rw-
start_brk 0x0000000000000000
end_code 0x00000000004009ec
start_code 0x0000000000400000
start_data 0x0000000000410d68
end_data 0x0000000000411030
start_stack 0x00000040007ff7b0
brk 0x0000000000411038
entry 0x00000040008111c0
argv_start 0x00000040007ff7b8
env_start 0x00000040007ff7c8
auxv_start 0x00000040007ff9a0
Hello World...

[Peter Maydell (pmaydell)]

If you objdump the binary and the offending library what do they seem to have been built for ?

Certainly this:

0000004000000000-0000004000001000 0000000000001000 ---

looks like a 4K page when we're trying to load things, so either we got the loading wrong or the binary is 4K.

[Alex Bennée (ajbennee)]

Do binaries have to be page size aware? I thought it was a runtime thing.
However if the aarch64-linux-user is hardwired to 4k it might explain it's
confusion on a 64k machine.

On Thu, 28 Nov 2019, 16:33 Peter Maydell, <email address hidden> wrote:

> If you objdump the binary and the offending library what do they seem to
> have been built for ?
>
> Certainly this:
>
> 0000004000000000-0000004000001000 0000000000001000 ---
>
> looks like a 4K page when we're trying to load things, so either we got
> the loading wrong or the binary is 4K.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1853826
>
> Title:
> ELF loader fails to load shared object on ThunderX2 running RHEL7
>
> Status in QEMU:
> Incomplete
>
> Bug description:
> Simple test:
> hello.c
>
> include <stdio.h>
>
> int main(int argc, char* argv[])
> {
> {
> printf("Hello World... \n");
> }
> return 0;
> }
>
> when compiled with :
> *Compiler
>
> https://developer.arm.com/tools-and-software/server-and-hpc/arm-architecture-tools/arm-allinea-studio/download
> Arm-Compiler-for-HPC_19.3_RHEL_7_aarch64.tar
>
> *Running:
> 1) with -armpl
> armclang -armpl hello.c
> ./qemu/build/aarch64-linux-user/qemu-aarch64 a.out
> 2) without flag
> armclang hello.c
> ./qemu/build/aarch64-linux-user/qemu-aarch64 a.out
>
> •With Docker image:
> CentOS Linux release 7.7.1908 (AltArch)
>
> *Two different machines:
> AArch64, Taishan. tsv110, Kunpeng 920, ARMv8.2-A
> AArch64, Taishan 2280, Cortex-A72, ARMv8-A
>
> *QEMU 4.0
> qemu-aarch64 version 4.1.91 (v4.2.0-rc1)
>
>
> Results:
>
>
> ****Taishan 2280 Cortex-A72
> Running
> 1)with -armpl flag with and without the docker
> WORKS-> Hello World...
> -> ldd a.out
> ldd a.out
> linux-vdso.so.1 => (0x0000ffffbc6a2000)
> libamath_generic.so =>
> /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libamath_generic.so
> (0x0000ffffbc544000)
> libm.so.6 => /lib64/libm.so.6 (0x0000ffffbc493000)
> libastring_generic.so =>
> /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libastring_generic.so
> (0x0000ffffbc472000) libarmflang.so =>
> /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libarmflang.so
> (0x0000ffffbbfd3000)
> libomp.so =>
> /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/libomp.so
> (0x0000ffffbbef5000)
> librt.so.1 => /lib64/librt.so.1 (0x0000ffffbbed4000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x0000ffffbbe9f000)
> libarmpl_lp64_generic.so =>
> /scratch/arm-linux-compiler-19.3_Generic-AArch64_RHEL-8_aarch64-linux/lib/clang/9.0.1/armpl_links/lib/libarmpl_lp64_generic.so
> (0x0000ffffb3306000)
> libc.so.6 => /lib64/libc.so.6 (0x0000ffffb3180000)
> libstdc++.so.6 =>
> /scratch/gcc-9.2.0_Generic-AArch64_RHEL-8_aarch64-linux/lib64/libstdc++.so.6
> (0x0000ffffb2f30000)
> libgcc_s.so.1 =>
> /scratch/gcc-9.2.0_Generic-...

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]