eztrace FTBFS with LTO

[Graham Inggs]

eztrace 2.0+repack-11 FTBFS due to mpich test failure when built with LTO enabled.

The following tests FAILED:
   3 - mpi_tests (Failed)
   5 - memory_tests (Failed)
  10 - openmp_tests (Failed)
  12 - pthread_tests (Failed)
  15 - posixio_tests (Failed)
Errors while running CTest
make[2]: *** [Makefile:74: test] Error 8
make[2]: Leaving directory '/<<PKGBUILDDIR>>/build-mpich'

The mpich tests pass after disaling LTO, however then the ompt tests still fail.

The following tests FAILED:
   5 - ompt_tests (Failed)
Errors while running CTest
make[2]: *** [Makefile:74: test] Error 8
make[2]: Leaving directory '/<<PKGBUILDDIR>>/build-ompt'

[Lukas Märdian (slyon)]

As Graham stated, disabling LTO make the build/tests (mostly) pass, but fails for the "OMPT_ARCHS" (amd64 arm64 i386 ppc64el riscv64), as defined in d/rules.

This can be done via lto-disabled-list or d/rules:
export DEB_BUILD_MAINT_OPTIONS = hardening=+all optimize=-lto

I tried a few different things in eztrace to make the toolchain mimic the Debian build (which passes):
* Switch to clang-14:
  CC=clang-14 in d/rules and clang-14 & libomp-14-dev B-Ds in d/control
* Strip Ubuntu's -Bsymbolic-functions optimization from the linker flags
  export DEB_LDFLAGS_MAINT_STRIP = -Wl,-Bsymbolic-functions in d/rules

But didn't have any success with the above changes.

Then I noticed bug #1899199, which indicates the root-cause being in the libomp5 dependency instead of eztrace itself, i.e. llvm-toolchain-* (or libomp.so.5 specifically) needs to be build without the -Wl,-Bsymbolic-functions LDFLAGS optimization. This seems to be the case!

After downloading relevant binaries from Debian (https://packages.debian.org/source/sid/llvm-toolchain-15), I was able to build eztrace successfully (with just LTO disabled, but no other changes):

sbuild --extra-package ../libomp5-15_15.0.7-4_amd64.deb --extra-package ../libomp-15-dev_15.0.7-4_amd64.deb -dlunar .

So IMO this is something for the Foundations toolchain people to look into, fixing our llvm-toolchain (libomp) build.

[François Trahay (trahay)]

I'm the main developer of EZTrace. The OMPT problem does come from bug #1899199.

If the llvm-toolchain build is not fixed in Ubuntu, I can implement a workaround in EZTrace (as suggested in https://bugs.launchpad.net/ubuntu/+source/llvm-defaults/+bug/1899199/comments/11), but I would prefer to avoid such unnecessary hacks.

[Lukas Märdian (slyon)]

François,
If you could come up with a patch (fix/workaround) for EZTrace, that'd be much appreciated!
Applying the "-Wl,-Bsymbolic-functions" fix to llvm-toolchain will most probably lead to a bigger discussion, as this is an optimization Ubuntu enabled on purpose a long time ago, and disabling it in a core-package like the LLVM toolchain might be a high impact change.

See this email thread for the -Wl,-Bsymbolic-functions background:
https://lists.ubuntu.com/archives/ubuntu-devel/2018-June/040369.html

[Launchpad Janitor (janitor)]

This bug was fixed in the package lto-disabled-list - 40

---------------
lto-disabled-list (40) devel; urgency=medium

  * Add clasp on s390x to avoid FTBFS (LP: #2017091)
  * Add eztrace to avoid FTBFS (LP: #2016471)

 -- Lukas Märdian <email address hidden> Thu, 20 Apr 2023 09:23:56 +0200

[Steve Langasek (vorlon)]

> Applying the "-Wl,-Bsymbolic-functions" fix to llvm-toolchain
> will most probably lead to a bigger discussion, as this is an
> optimization Ubuntu enabled on purpose a long time ago,
> and disabling it in a core-package like the LLVM toolchain
> might be a high impact change.

In fact, there are almost no reverse-dependencies of libomp in Ubuntu. I've uploaded a patch that disables -Bsymbolic-functions for libomp only, which should hopefully fix the eztrace build.

[Steve Langasek (vorlon)]

Attempted to rebuild eztrace with the libomp built with -Wl,-Bsymbolic and it didn't help.

Correct version of libomp downloaded:

Get:108 http://ftpmaster.internal/ubuntu mantic-proposed/universe amd64 libomp5-15 amd64 1:15.0.7-4ubuntu2 [480 kB]

(https://launchpad.net/ubuntu/+source/eztrace/2.0+repack-12/+build/26015911)

expected linker options when building libomp:

[2100/2653] : && /<<PKGBUILDDIR>>/build-llvm/./bin/clang --target=x86_64-pc-linux-gnu -fPIC -fstack-protector-strong -Wformat -Werror=format-security -Wno-unused-command-line-argument -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -fno-semantic-interposition -Werror=date-time -Werror=unguarded-availability-new -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wmissing-field-initializers -Wimplicit-fallthrough -Wcovered-switch-default -Wstring-conversion -Wmisleading-indentation -fdiagnostics-color -ffunction-sections -fdata-sections -fdebug-prefix-map=/<<PKGBUILDDIR>>/build-llvm/runtimes/runtimes-bins=build-llvm/runtimes/runtimes-bins -fdebug-prefix-map=/<<PKGBUILDDIR>>/= -no-canonical-prefixes -ffile-prefix-map=/<<PKGBUILDDIR>>/build-llvm/runtimes/runtimes-bins=build-llvm/runtimes/runtimes-bins -ffile-prefix-map=/<<PKGBUILDDIR>>/= -no-canonical-prefixes -Wall -Wcast-qual -Wformat-pedantic -Wimplicit-fallthrough -Wsign-compare -Wno-extra -Wno-pedantic -O3 -DNDEBUG -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,--build-id -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=gold -Wl,--warn-shared-textrel -Wl,--as-needed -Wl,--version-script=/<<PKGBUILDDIR>>/openmp/runtime/src/exports_so.txt -static-libgcc -Wl,-z,noexecstack -Wl,-Bno-symbolic -shared -Wl,-soname,libomp.so.5 -o openmp/runtime/src/libomp.so.5 openmp/runtime/src/CMakeFiles/omp.dir/kmp_alloc.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_atomic.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_csupport.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_debug.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_itt.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_environment.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_error.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_global.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_i18n.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_io.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_runtime.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_settings.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_str.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_tasking.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_threadprivate.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_utility.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_barrier.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_wait_release.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_affinity.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_dispatch.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_lock.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_sched.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/z_Linux_util.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_gsupport.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/thirdparty/ittnotify/ittnotify_static.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_taskdeps.cpp.o openmp/runtime/src/CMakeFiles/omp.dir/kmp_cancel.c...

[Steve Langasek (vorlon)]

Looking more closely, there's no relocation for this symbol because the symbol isn't present at all.
$ objdump -T ./build-llvm/runtimes/runtimes-bins/openmp/runtime/src/libomp.so.5 | grep -i ompt
$

I'm not sure if this is a result of a deliberate change in the llvm implementation, but it certainly doesn't help to build with -Bno-symbolic if the symbol doesn't exist.

[Graham Inggs (ginggs)]

This appears to be fixed now in:

llvm-toolchain-15 (1:15.0.7-4ubuntu6) mantic; urgency=medium

  * Fix up -Wl,-Bsymbolic-functions disable
  * Cherry-pick upstream fix for cmake changes

[Launchpad Janitor (janitor)]

This bug was fixed in the package llvm-toolchain-15 - 1:15.0.7-6

---------------
llvm-toolchain-15 (1:15.0.7-6) unstable; urgency=medium

  [ Sylvestre Ledru ]
  * Only pass -Bno-symbolic if it exists

  [ Gianfranco Costamagna ]
  * Revert LIBCXXABI_ARM_EHABI => _LIBCXXABI_ARM_EHABI change,
    FTBFS on arm*.

 -- Sylvestre Ledru <email address hidden> Mon, 26 Jun 2023 10:13:40 +0200

[François Trahay (trahay)]

The problem was fixed in the llvm-toochain-15 package, but I also fixed it in the eztrace (upstream) repo.

EZTrace version 2.1 now uses `OMP_TOOL_LIBRARIES=$LD_PRELOAD` as a workaround, as suggested in
https://bugs.launchpad.net/ubuntu/+source/llvm-defaults/+bug/1899199/comments/11