FTBFS with llvm-10

[Christian Ehrhardt ]

Reported by Doko.
https://launchpadlibrarian.net/468757374/buildlog_ubuntu-focal-s390x.postgresql-12_12.2-1ubuntu1_BUILDING.txt.gz

Reproduces when built in a local system.

[Christian Ehrhardt  (paelzer)]

Log file out of the build process that fails.
At the end is the command to /usr/lib/llvm-10/bin/llvm-lto that crashes

#0 0x000003ff7faab14a (/usr/lib/s390x-linux-gnu/libLLVM-10.so.1+0x9ab14a)
Segmentation fault

[Christian Ehrhardt  (paelzer)]

It doesn't need all the arguments and can be shortened to:
/usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc
Stack dump:
0. Program arguments: /usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc
#0 0x000003ff9e32b14a (/usr/lib/s390x-linux-gnu/libLLVM-10.so.1+0x9ab14a)
Segmentation fault

[Christian Ehrhardt  (paelzer)]

Here a backtrace in GDB, let me know if I can check anything for you there:

ubuntu@s1lp05:~/postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode$ gdb /usr/lib/llvm-10/bin/llvm-ltoGNU gdb (Ubuntu 9.1-0ubuntu1) 9.1
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "s390x-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/llvm-10/bin/llvm-lto...
Reading symbols from /usr/lib/debug/.build-id/2c/dfd421eef6cbe0600d24e01b38e71763bb4333.debug...
(gdb) run -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc
Starting program: /usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/s390x-linux-gnu/libthread_db.so.1".

Program received signal SIGSEGV, Segmentation fault.
handleErrors<(lambda at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:988:33)>(void) ()
    at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:907
907 /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h: No such file or directory.
(gdb) bt
#0 handleErrors<(lambda at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:988:33)>(void) ()
    at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:907
#1 0x000003fff9d512e0 in handleAllErrors<(lambda at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:988:33)> ()
    at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:925
#2 toString () at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/include/llvm/Support/Error.h:988
#3 0x000003fffab16536 in addModule () at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/lib/LTO/ThinLTOCodeGenerator.cpp:515
#4 0x000000008001c5ce in thinLink () at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/tools/llvm-lto/llvm-lto.cpp:577
#5 0x000000008001464a in run () at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/tools/llvm-lto/llvm-lto.cpp:541
#6 main () at /build/llvm-toolchain-10-PvK6ID/llvm-toolchain-10-10.0.0~+rc3/llvm/tools/llvm-lto/llvm-lto.cpp:912

[Christian Ehrhardt  (paelzer)]

Attaching a core dump out of gdb matching this issue.

[Jonas Paulsson (jonpsson)]

I would be happy to help on this but not sure where to start - to run

/usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc

, I would need the input brin.bc file, but I don't have it...

[Christian Ehrhardt  (paelzer)]

This is the full build directory.

to reach the error in there do:
$ postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode
$ /usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc

I extracted the same on x86 and tried it there, it works on x86.
So you really will need an s390x system to trigger the crash.

[Christian Ehrhardt  (paelzer)]

Note: running the same against llvm-9 fails (probably because the former build steps were done with llvm-10)

/usr/lib/llvm-9/bin/llvm-lto -thinlto -thinlto-action=thinlink -o postgres.index.bc postgres/access/brin/brin.bc
error: can't create module summary index for buffer: Invalid summary version 8. Version should be in the range [1-7].
LLVM ERROR: ThinLink didn't create an index

But we know that with llvm-9 the full build works. Never the less that will make e.g. bisecting harder as we can't just repeat this last step.

[Jonas Paulsson (jonpsson)]

I tried it on trunk (9adc7fc) built with gcc, and then also built a second time with itself, but it did not show up.

It would help me if I could get the right cmake line, and the git commit ID. I used:

cmake -G Ninja -DLLVM_ENABLE_PROJECTS="clang;compiler-rt" -DCMAKE_BUILD_TYPE="Release" -DLLVM_ENABLE_ASSERTIONS=On

Not sure what to try next - maybe disabling assertions and only building the SystemZ target?

[Christian Ehrhardt  (paelzer)]

llvm in Focal is currently at:
10.0.0~+rc5-1ubuntu1

Which should roughly be the RC5 tag plus d21664cce1db8debe2528f36b1fbd2b8af9c9401

On [1] you can find produced binaries, also the sources if you extract the tarballs and the build logs.

I think the cmake call is the following, but please have a look at the build log yourself if in doubt.

cd build-llvm && \
 $CMAKE_BIN ../llvm/ \
-DLLVM_ENABLE_PROJECTS="clang;clang-tools-extra;libcxx;libcxxabi;compiler-rt;lld;debuginfo-tests;polly;openmp;lldb" \
-DCMAKE_INSTALL_PREFIX=/usr/lib/llvm-10 \
-DCMAKE_VERBOSE_MAKEFILE=ON \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_CXX_FLAGS_RELWITHDEBINFO="-O2 -DNDEBUG -g1" \
-DLLVM_LINK_LLVM_DYLIB=ON \
-DLLVM_INSTALL_UTILS=ON \
-DLLVM_VERSION_SUFFIX= \
-DLLVM_ENABLE_SPHINX=ON \
-DSPHINX_WARNINGS_AS_ERRORS=OFF \
-DLLVM_BUILD_LLVM_DYLIB=ON \
-DLLVM_ENABLE_RTTI=ON \
-DLLVM_ENABLE_FFI=ON \
-DPYTHON_EXECUTABLE=/usr/bin/python3 -DLLVM_BINUTILS_INCDIR=/usr/include/ -DLLVM_USE_PERF=yes -DLLVM_ENABLE_LIBPFM=ON \
-DLIBCLANG_LIBRARY_VERSION=1 \
-DCLANG_LINK_CLANG_DYLIB=ON \
-DENABLE_LINKER_BUILD_ID=ON \
-DPOLLY_BUNDLED_JSONCPP=OFF \
-DCOMPILER_RT_USE_LIBCXX=NO \
-DLLVM_ENABLE_Z3_SOLVER=OFF \
-DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="AVR" \
-DLLVM_TEMPORARILY_ALLOW_OLD_TOOLCHAIN=ON \
-DCLANG_VENDOR=Ubuntu \
-DCLANG_ENABLE_BOOTSTRAP=ON \
-DCLANG_BUILD_EXAMPLES=OFF \
-DBOOTSTRAP_CMAKE_CXX_FLAGS=' -fuse-ld=gold -fPIC -Wno-unused-command-line-argument -Wno-unknown-warning-option ' \
-DBOOTSTRAP_CMAKE_C_FLAGS=' -fuse-ld=gold -fPIC -Wno-unused-command-line-argument -Wno-unknown-warning-option ' \
-DCLANG_BOOTSTRAP_PASSTHROUGH="CMAKE_INSTALL_PREFIX;CMAKE_VERBOSE_MAKEFILE;CMAKE_BUILD_TYPE;CMAKE_CXX_FLAGS_RELWITHDEBINFO;LLVM_LINK_LLVM_DYLIB;LLVM_INSTALL_UTILS;LLVM_VERSION_SUFFIX;LLVM_ENABLE_SPHINX;SPHINX_WARNINGS_AS_ERRORS;LLVM_BUILD_LLVM_DYLIB;LLVM_ENABLE_RTTI;LLVM_ENABLE_FFI;LIBCLANG_LIBRARY_VERSION;ENABLE_LINKER_BUILD_ID;POLLY_BUNDLED_JSONCPP;LLVM_EXPERIMENTAL_TARGETS_TO_BUILD;LLVM_USE_PERF;LLVM_ENABLE_ASSERTIONS;LLVM_BINUTILS_INCDIR;LLVM_HOST_TRIPLE;LLVM_COMPILER_CHECKED;COMPILER_RT_BUILD_BUILTINS;LIBOMP_LIBFLAGS;CMAKE_SHARED_LINKER_FLAGS;PYTHON_EXECUTABLE;LLVM_ENABLE_Z3_SOLVER"

[1]: https://launchpad.net/ubuntu/+source/llvm-toolchain-10/1:10.0.0~+rc5-1ubuntu1/+build/18857669

[Jonas Paulsson (jonpsson)]

I tracked this down to a single commit that if reverted stops the test case from segfaulting: 2d5820c. This is a patch that improves CorrelatedValuePropagation to also handle And instructions.

I am currently working on finding the exact bug - I am not yet sure if the bug is actually in that pass or if this change merely exposes something else.

One thing I tried was to insert a clearing of the LazyValueInfo cache before each call to it. This changed the results somewhat, but llvm-lto still crashes.

[Jonas Paulsson (jonpsson)]

I think I found a bug that I would guess is causing this crash:

https://bugs.llvm.org/show_bug.cgi?id=45384

[Ulrich Weigand (uweigand)]

It seems this was a known common-code issue that is fixed in mainline not in LLVM 10. There's already a bug to port it to 10.0.1 (PR45272) with a back ported patch https://reviews.llvm.org/D76596.

Can you try whether this fixes the problem?

[Jonas Paulsson (jonpsson)]

I tried it, and it seems to fix this test case at least.

[Christian Ehrhardt  (paelzer)]

Old version:
ubuntu@s1lp05:~$ dpkg -S /usr/lib/llvm-10/bin/llvm-lto
llvm-10: /usr/lib/llvm-10/bin/llvm-lto
ubuntu@s1lp05:~$ dpkg -l llvm-10
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-======================-============-===========================================
ii llvm-10 1:10.0.0~+rc5-1ubuntu1 s390x Modular compiler and toolchain technologies

Fails:
ubuntu@s1lp05:~$ /usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o /home/ubuntu/postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode/postgres.index.bc /home/ubuntu/postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode/postgres/access/brin/brin.bc
Stack dump:
0. Program arguments: /usr/lib/llvm-10/bin/llvm-lto -thinlto -thinlto-action=thinlink -o /home/ubuntu/postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode/postgres.index.bc /home/ubuntu/postgresql-12-12.2/build/tmp_install/usr/lib/postgresql/12/lib/bitcode/postgres/access/brin/brin.bc
#0 0x000003ffa032b1ca PrintStackTrace /build/llvm-toolchain-10-m0H4U3/llvm-toolchain-10-10.0.0~+rc5/llvm/lib/Support/Unix/Signals.inc:568:7
#1 0x000003ffa032b1ca PrintStackTraceSignalHandler /build/llvm-toolchain-10-m0H4U3/llvm-toolchain-10-10.0.0~+rc5/llvm/lib/Support/Unix/Signals.inc:624:3
Segmentation fault

Upgrade:
ubuntu@s1lp05:~$ sudo apt dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  libapt-pkg5.90 libberkeleydb-perl libbind9-161 libboost-iostreams1.67.0 libboost-system1.67.0 libboost-thread1.67.0 libbrlapi0.6 libcbor0 libcgi-fast-perl libcgi-pm-perl
  libclass-accessor-perl libdns-export1104 libdns-export1107 libdns1104 libdns1107 libdns1109 libfcgi-perl libfreezethaw-perl libicu63 libirs161 libisc-export1100 libisc-export1104
  libisc1100 libisc1104 libisc1105 libisccc161 libisccfg163 liblwres161 libmldbm-perl libprocps7 libpython2-stdlib libpython2.7-minimal libpython2.7-stdlib libpython3.7
  libpython3.7-minimal libpython3.7-stdlib libruby2.5 libsbc1 libspandsp2 libwireshark12 libwiretap9 libwscodecs2 libwsutil10 linux-5.4-headers-5.4.0-14 linux-headers-4.15.0-62
  linux-headers-4.15.0-62-generic linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic linux-headers-5.4.0-14-generic linux-headers-5.4.0-9 linux-headers-5.4.0-9-generic
  linux-image-4.15.0-62-generic linux-image-4.15.0-74-generic linux-image-5.4.0-14-generic linux-image-5.4.0-9-generic linux-modules-4.15.0-62-generic linux-modules-4.15.0-74-generic
  linux-modules-5.4.0-14-generic linux-modules-5.4.0-9-generic linux-modules-extra-4.15.0-62-generic linux-modules-extra-4.15.0-74-generic linux-modules-extra-5.4.0-14-generic
  linux-modules-extra-5.4.0-9-generic php7.3-dev python2 python2-minimal python2.7 python2.7-minimal python3-asn1crypto python3-nacl python3-pymacaroons python3.7 python3.7-minimal ruby2.5
...

[Launchpad Janitor (janitor)]

This bug was fixed in the package llvm-toolchain-10 - 1:10.0.0-1ubuntu2

---------------
llvm-toolchain-10 (1:10.0.0-1ubuntu2) focal; urgency=medium

  * Apply D76596.diff, proposed backport from the trunk. LP: #1867173.

 -- Matthias Klose <email address hidden> Thu, 02 Apr 2020 11:56:42 +0200

[Seth Arnold (seth-arnold)]

Hello, I don't quickly spot why this bug is marked security; is there a security component here that I am overlooking?

Thanks

[Christian Ehrhardt  (paelzer)]

Hi Seth,
I see no security aspect either.

Ulrich marked it as such, maybe he can speak up.
Otherwise I think this is fine to be considered a non-security issue.

[Ulrich Weigand (uweigand)]

Huh, sorry. Must have been a mis-click, I just wanted to subscribe to the bug I think. I've removed the flag now.