[UBUNTU 23.04] S390: static-PIE programs segfaults if libc6-dev-s390x-cross package is installed

[bugproxy]

== by Stefan <email address hidden> ==
A simple helloworld program build and linked as static-PIE segfaults while startup in __libc_setup_tls:
$ gcc -c -fPIE -static-pie -o hello.o hello.c
$ gcc -o hello hello.o -static-pie

Note:
If only libc6-dev package is installed, all is fine.
If both libc6-dev and libc6-dev-s390x-cross packages are installed, you will see the mentioned segfault.

Linking with "-Wl,-v" dumps the used linker command and it shows the used startup-files:
/usr/lib/s390x-linux-gnu/rcrt1.o => from libc6-dev
/usr/s390x-linux-gnu/lib/crti.o => from libc6-dev-s390x-cross
/usr/lib/gcc/s390x-linux-gnu/12/crtbeginS.o
/usr/lib/gcc/s390x-linux-gnu/12/crtendS.o
/usr/s390x-linux-gnu/lib/crtn.o => from libc6-dev-s390x-cross

Linking as static-PIE requires the rcrt1.o file, which is not available in libc6-dev-s390x-cross package, but in libc6-dev. Due to this mixing of the startup-files, you get the segfault.

This issue can be fixed by enabling static-PIE also in the libc6-dev-s390x-cross package, then all startup-files belong to the same package. For s390x static-PIE was introduced in glibc 2.36:
commit "S390: Enable static PIE" (in glibc 2.36)
https://sourceware.org/git/?p=glibc.git;a=commit;h=728894dba4a19578bd803906de184a8dd51ed13c

There is a configure check which do a link-test to ensure that a suitable binutils(ld) version is used. Afterwards static-PIE is automatically enabled. The required binutils-patches are first included in binutils 2.39.

According to the build log of package cross-toolchain-base (see https://launchpad.net/ubuntu/+source/cross-toolchain-base/66ubuntu3/+build/25689036), the libc6-dev-s390x-cross package is cross-build on x86_64 and the mentioned configure check fails:
running configure fragment for sysdeps/s390/s390-64
checking for s390-specific static PIE requirements... no

In this cross build, glibc is configured in order to first build the crt-startup-files, which are needed to complete the cross-gcc build. At this time, the sysroot does not contain the crt-files or libc.so itself. Thus the "linking" configure check is failing. After building the cross-gcc, glibc is build without re-configuring. Thus static-PIE is not enabled.

In glibc-upstream, this configure check is now adjusted and it allows checking binutils by version number:
commit "s390x: Fix static PIE condition for toolchain bootstrapping." (will be in glibc 2.39)
https://sourceware.org/git/?p=glibc.git;a=commit;h=f5f96b784beb3480e0e8d10e250ca7e6063ab881

Perhaps you also have to pick the following commits by Sam James which adjusted the tests in between (both are in glibc 2.36):
- commit "s390: use $READELF"
https://sourceware.org/git/?p=glibc.git;a=commit;h=c376ff3287b9b0f78a4f8951313c6dae60cbdfea
- commit "s390: use LC_ALL=C for readelf call"
https://sourceware.org/git/?p=glibc.git;a=commit;h=2249ec60a987f9a7aa585890de2bd365b3656d28

In addition to the static-PIE configure-checks, there are those other s390-specific configure-checks to determine which IFUNC-optimizations can be build and used as default. Those also fail for libc6-dev-s390x-cross as linking is also involved:
running configure fragment for sysdeps/s390
checking for __builtin_tbegin... yes
checking for S390 vector instruction support... no
configure: WARNING: Use binutils with vector-support in order to use optimized implementations.
checking for S390 vector support in gcc... no
checking for S390 arch13 zarch instruction support... no
checking for S390 z10 zarch instruction support as default... no
checking for S390 z196 zarch instruction support as default... no
checking for S390 z13 zarch instruction support as default... no
checking for S390 arch13 zarch instruction support as default... no

Those checks were also adjusted in glibc-upstream. Please also pick this commits:
commit "S390: Use compile-only instead of also link-tests in configure." (in glibc 2.38)
https://sourceware.org/git/?p=glibc.git;a=commit;h=368b7c614b102122b86af3953daea2b30230d0a8

I've observed this issue on Ubuntu 23.04 with glibc 2.37 and binutils 2.40:
binutils/lunar-updates,lunar-security,now 2.40-2ubuntu4.1 s390x [installed]
libc6-dev-s390x-cross/lunar,now 2.37-0ubuntu2cross1 all [installed]
libc6-dev/lunar,now 2.37-0ubuntu2 s390x [installed]
libc6-s390x-cross/lunar,now 2.37-0ubuntu2cross1 all [installed,automatic]
libc6/lunar,now 2.37-0ubuntu2 s390x [installed]

I think Ubuntu 22.10 is already out of support and Ubuntu 22.04 has no support for static-PIE on s390x as using too old glibc 2.35 and binutils 2.38.

[Matthias Klose (doko)]

what is the rationale of having the libc6-dev-s390x-cross package installed on s390x?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-23 07:39 EDT-------
Hi,

it was basically by chance because we ported a software project from x86 to s390x
and the original project used these cross packages to build and run (unit)tests for non-x86 architectures with qemu-user.

[Matthias Klose (doko)]

we probably could have a conflict in libc6-dev:

Conflicts: libc6-dev-s390x-cross [s390x]

Or figure out what is wrong with the search path. maybe avoid the lookups into /usr/s390x-linux-gnu, when --sysroot=/ is specified

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-08-24 07:03 EDT-------
(In reply to comment #9)
> we probably could have a conflict in libc6-dev:
>
> Conflicts: libc6-dev-s390x-cross [s390x]
Interesting. I don't see this conflict with "apt info libc6-dev".

>
> Or figure out what is wrong with the search path. maybe avoid the lookups
> into /usr/s390x-linux-gnu, when --sysroot=/ is specified
The search path includes both, the path where the cross-startup-files are stored and the non-cross ones. If something like this is changed, it should be done for all architectures.

Nevertheless, if somebody e.g. on x86_64 uses libc6-dev-s390x-cross, then static-PIE is not available. Thus the configure-checks should be fixed anyway.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-14 04:02 EDT-------
Hi doko,
are there any news regarding the cross packages?

[Matthias Klose (doko)]

that should be fixed in mantic (23.10).

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2023-11-27 04:25 EDT-------
I've just upgraded to mantic (23.10) and I still see that linking static PIE programs leads to a segfaults at runtime.
As mentioned in the bug-report, if both libc6-dev and libc6-dev-s390x-cross packages are installed, the crt-object files from the cross package are prefered and as the cross package still lacks rcrt1.o (needed for static PIE), it just uses it from the non-cross package. Due to the mixture of crt-files, it ends up in the segfault at runtime.

Here is the list of my installed libc6-packages:
# apt list --installed "libc6*"
Listing... Done
libc6-dbg/mantic,now 2.38-1ubuntu6 s390x [installed]
libc6-dev-s390/mantic,now 2.38-1ubuntu6 s390x [installed]
libc6-dev-s390x-cross/mantic,now 2.38-1ubuntu4cross1 all [installed]
libc6-dev/mantic,now 2.38-1ubuntu6 s390x [installed]
libc6-s390/mantic,now 2.38-1ubuntu6 s390x [installed]
libc6-s390x-cross/mantic,now 2.38-1ubuntu4cross1 all [installed,automatic]
libc6/mantic,now 2.38-1ubuntu6 s390x [installed]

[Matthias Klose (doko)]

ok, so the missing rcrt1.o file looks like another issue. we probably should revert to the two-staged builds, if that configuration isn't picked up correctly.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-02-02 04:24 EDT-------
are there any news regarding missing rcrt1.o?

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-02-26 03:48 EDT-------
Ubuntu noble 24.04 is using glibc 2.39 which includes the required patch:
> In glibc-upstream, this configure check is now adjusted and it allows checking binutils by version number:
> commit "s390x: Fix static PIE condition for toolchain bootstrapping." (will be in glibc 2.39)
> https://sourceware.org/git/?p=glibc.git;a=commit;h=f5f96b784beb3480e0e8d10e250ca7e6063ab881

The libc6-dev-s390x-cross package now contains the rcrt1.o file:
https://packages.ubuntu.com/noble/all/libc6-dev-s390x-cross/filelist
/usr/s390x-linux-gnu/lib/rcrt1.o

I've also upgraded to current pre-release of Ubuntu 24.04 and checked that compiling/running a static pie program now works. Linking with "-Wl,-v" also shows that rcrt1.o is now also used from libc6-dev-s390x-cross package.