arm64: broken c++ exception handler support leads to std::terminate() being called and program abort

[Ali Saidi]

[Impact]

On architectures other than i386 and amd64, the C++ exception support in libunwind appears to be broken, always failing and calling std::terminate() which leads to the program aborting.

(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x0000fffff7c2daac in __GI_abort () at abort.c:79
#2 0x0000fffff7e21868 in __gnu_cxx::__verbose_terminate_handler() ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#3 0x0000fffff7e1f21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4 0x0000fffff7e1f280 in std::terminate() ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#5 0x0000fffff7e1f5e0 in __cxa_rethrow ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#6 0x0000fffff7e21804 in __gnu_cxx::__verbose_terminate_handler() ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#7 0x0000fffff7e1f21c in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#8 0x0000fffff7e1f280 in std::terminate() ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#9 0x0000fffff7e1f574 in __cxa_throw ()
   from /lib/aarch64-linux-gnu/libstdc++.so.6
#10 0x0000fffff7fb9f50 in function_throws_int () at lib.cpp:9
#11 0x0000aaaaaaaa0d54 in main (argc=1, argv=0xfffffffffab8) at main.cpp:9

Compiling libunwind with --enable-cxx-exceptions enabled leads to _Unwind_RaiseException being called during __cxa_throw(), which fails to find a handler, and the generic std::terminate() is called instead, aborting the program.

On i386 and amd64 this doesn't seem to be the case, and the libunwind handlers seem to be present.

To fix, we only enable the configure option --enable-cxx-exceptions on i386 and amd64 only, in debian/rules. This lets other architectures fall back to the symbols provided by libgcc_s, which implementation works correctly.

[Testcase]

Ali Sadi has provided a reproducer program.

Start an arm64 instance, for example, a c6g.medium instance on AWS, with either Bionic or Focal.

$ wget https://bugs.launchpad.net/ubuntu/+source/libunwind/+bug/1999104/+attachment/5635122/+files/libunwind.tar.gz
$ sudo apt install -y build-essential libunwind-dev
$ tar xvf libunwind.tar.gz && cd test
$ make all

There are two executable, main and main_unwind. main is not linked to libunwind, and main_unwind is linked to libunwind.

$ ./main
int throws lib
int caught main
$ ./main_unwind
terminate called after throwing an instance of 'int'
terminate called recursively
Aborted (core dumped)

If you install the test package available in the following ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/sf350246-test

$ make clean
$ sudo apt install -y libunwind-dev
$ make all
$ ./main
int throws lib
int caught main
$ ./main_unwind
int throws lib
int caught main

The exception is caught as expected the program does not abort.

[Where problems could occur]

For architectures other than i386 and amd64, we are changing from libunwind provided exception handlers for __cxa_throw(), and using those provided by libgcc_s instead.

There are a few reverse dependencies for libunwind-dev and libunwind8, which need to be considered:

$ apt rdepends libunwind-dev
libunwind-dev
Reverse Depends:
  Depends: libunwind-setjmp0-dev (= 1.2.1-9build1)
  Depends: libefl-all-dev

t$ apt rdepends libunwind-dev 8
libunwind8
Reverse Depends:
  Depends: libunwind-dev (= 1.2.1-9build1)
  Depends: xvfb
  Depends: xnest
  Depends: xdmx
  Depends: xwayland
  Depends: xserver-xorg-core
  Depends: xserver-xephyr
  Depends: linux-tools-5.4.0-*
  Depends: linux-raspi-tools-*
  Depends: linux-raspi2-tools-5.4.0-*
  Depends: linux-raspi2-5.4-tools-5.4.0-*
  Depends: linux-oracle-5.15-tools-5.15.0-*
  Depends: linux-lowlatency-hwe-5.15-tools-5.15.0-*
  Depends: linux-hwe-5.8-tools-5.8.0-*
  Depends: linux-hwe-5.15-tools-5.15.0-*
  Depends: linux-gke-tools-5.4.0-*
  Depends: linux-gke-5.15-tools-5.15.0-*
  Depends: linux-gcp-tools-5.4.0-*
  Depends: linux-gcp-5.15-tools-5.15.0-*
  Depends: linux-azure-tools-5.4.0-*
  Depends: linux-azure-5.15-tools-5.15.0-*
  Depends: linux-aws-tools-5.4.0-*
  Depends: linux-aws-5.8-tools-5.8.0-*
  Depends: linux-aws-5.15-tools-5.15.0-*
  Depends: xvfb
  Depends: xnest
  Depends: xdmx
  Depends: trafficserver
  Depends: tilix
  Depends: tigervnc-standalone-server
  Depends: tarantool
  Depends: sysprof
  Depends: rspamd
  Depends: libwine-development
  Depends: libwine
  Depends: libjulia1
  Depends: libheaptrack
  Depends: libevas-loaders
  Depends: libephysics1
  Depends: libeina1a
  Depends: libecore-imf1
  Depends: julia
  Depends: geary
  Depends: gdnsd
  Depends: xwayland
  Depends: xserver-xorg-core
  Depends: xserver-xephyr
  Depends: libunwind-setjmp0

The reporter seems to be hitting the issue with mcrouter, built themselves, so we can likely skip no-change rebuilds of rdepends until we get actual complaints that a bug exists in those packages, to keep regression risk down.

This does of course, leave the risk of regression to the future, especially during critical times, e.g. CVE fix.

[Other info]

This was resolved in Debian bug 923962 by the maintainer, by setting the configure option --enable-cxx-exceptions only on i386 and amd64.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923962

The debdiff between Focal and Groovy is available here:

https://paste.ubuntu.com/p/8xv6FNPs8g/

[William Ashley (washley)]

I've slimmed down the reproducer to a single main function and removed any additional compiler arguments.

[William Ashley (washley)]

1.2.1-10 (https://launchpad.net/ubuntu/+source/libunwind/1.2.1-10) resolves this issue on 20. It includes a change to not use --enable-cxx-exceptions when building libunwind, and that is the key here. Even on Ubuntu 22, with its much newer upstream version of libunwind, adding that option triggers the same failures in C++ exceptions that we've observed here on 18 and 20.

[William Ashley (washley)]

The key difference that --enable-cxx-exceptions makes is that it results in libunwind having _Unwind_RaiseException (and likely other functions), which gets called during a C++ throw by __cxa_throw. Without libunwind, this symbol is provided by libgcc_s, whose implementation works correctly. For whatever reason, the one in libunwind, on aarch64, does not succeed in finding a handler and instead calls terminate (which is normal when there is no handler for an exception).

[Matthew Ruffell (mruffell)]

Attached is a debdiff which fixes this problem on Bionic

[Matthew Ruffell (mruffell)]

Attached is a debdiff which fixes this problem on Focal.

[Michael Hudson-Doyle (mwhudson)]

I tweaked the version numbers slightly (ubuntu1 -> ubuntu0.1) and uploaded.

[Timo Aaltonen (tjaalton)]

Hello Ali, or anyone else affected,

Accepted libunwind into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libunwind/1.2.1-8ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Timo Aaltonen (tjaalton)]

Hello Ali, or anyone else affected,

Accepted libunwind into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libunwind/1.2.1-9ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ali Saidi (alisaidi)]

Tested on focal with package libunwind8_1.2.1-9ubuntu0.1 and the reproducer ran successfully

[Ali Saidi (alisaidi)]

Tested on bionic with package libunwind8_1.2.1-8ubuntu0.1 and the reproducer ran successfully.

[Ali Saidi (alisaidi)]

Also tried folly (the dependency of mcrouter that was throwing the exception) and unit tests that previously failed because of this bug now pass.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted libunwind (1.2.1-8ubuntu0.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

linux-oracle-5.4/5.4.0-1091.100~18.04.1 (arm64)
linux-raspi2-5.3/blacklisted (arm64)
linux-hwe-5.0/blacklisted (armhf, s390x, arm64, ppc64el)
linux-hwe-5.4/5.4.0-136.153~18.04.1 (s390x, amd64, arm64, i386, ppc64el)
linux/4.15.0-201.212 (s390x, amd64, arm64, i386, ppc64el)
linux-gke-5.4/blacklisted (arm64)
linux-raspi-5.4/blacklisted (armhf, arm64)
linux-kvm/4.15.0-1133.138 (amd64)
linux-snapdragon/4.15.0-1143.153 (arm64)
linux-azure-5.4/5.4.0-1100.106~18.04.1 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#libunwind

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted libunwind (1.2.1-9ubuntu0.1) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-ibm/5.4.0-1041.46 (amd64)
linux-gcp-5.13/blacklisted (amd64)
linux-lowlatency-hwe-5.15/5.15.0-57.63~20.04.1 (arm64)
linux-intel-5.13/blacklisted (amd64)
linux-intel-iotg-5.15/5.15.0-1018.23~20.04.1 (amd64)
linux-gcp-5.8/blacklisted (amd64)
linux-oem-5.13/blacklisted (amd64)
linux-hwe-5.13/blacklisted (s390x, amd64, armhf, ppc64el, arm64)
linux-oracle-5.13/blacklisted (amd64, arm64)
linux-oem-5.10/blacklisted (amd64)
linux-gcp-5.11/blacklisted (amd64)
linux-oem-5.6/blacklisted (amd64)
linux-azure-5.13/blacklisted (arm64)
linux-azure-cvm/5.4.0-1100.106+cvm1 (amd64)
linux-azure-5.11/blacklisted (amd64, arm64)
linux-aws-5.13/blacklisted (amd64, arm64)
linux-oracle-5.15/5.15.0-1026.32~20.04.1 (amd64, arm64)
linux-hwe-5.8/blacklisted (s390x, amd64, ppc64el, arm64)
linux-gke-5.15/5.15.0-1023.28~20.04.2 (amd64, arm64)
linux-bluefield/5.4.0-1054.60 (arm64)
linux-gcp-5.15/5.15.0-1026.33~20.04.1 (arm64)
linux-hwe-5.11/blacklisted (s390x, amd64, armhf, ppc64el, arm64)
linux-oracle-5.11/blacklisted (amd64)
linux-aws-5.8/blacklisted (amd64)
linux-azure-5.8/blacklisted (amd64)
linux-aws-5.11/blacklisted (amd64, arm64)
linux-hwe-5.15/5.15.0-57.63~20.04.1 (s390x, amd64, ppc64el, arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#libunwind

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Matthew Ruffell (mruffell)]

Performing verification for Focal.

I started two instances on AWS, one c6g.medium (arm64) and a t2.micro (amd64).

I went through the reproducer listed in the testcase with libunwind-dev 1.2.1-9build1 from -release.

$ wget https://bugs.launchpad.net/ubuntu/+source/libunwind/+bug/1999104/+attachment/5635122/+files/libunwind.tar.gz
$ sudo apt install -y build-essential libunwind-dev
$ tar xvf libunwind.tar.gz && cd test
test/
test/lib.hpp
test/main.cpp
test/lib.cpp
test/Makefile
~/test$ make all
g++ -g -shared-libgcc -shared -fPIC -std=c++11 -o libtest.so lib.cpp
g++ -g -shared-libgcc -o main -L. -Wl,-rpath,. main.cpp -ltest
g++ -g -shared-libgcc -o main_unwind -L. -Wl,-rpath,. main.cpp -ltest -lunwind

On arm64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
terminate called after throwing an instance of 'int'
terminate called recursively
Aborted (core dumped)

On amd64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
int throws lib
int caught main

As expected, we see arm64 abort the execution of the reproducer.

I then installed 1.2.1-9ubuntu0.1 from -proposed and rebuilt the reproducers:

$ make clean
$ make all
g++ -g -shared-libgcc -shared -fPIC -std=c++11 -o libtest.so lib.cpp
g++ -g -shared-libgcc -o main -L. -Wl,-rpath,. main.cpp -ltest
g++ -g -shared-libgcc -o main_unwind -L. -Wl,-rpath,. main.cpp -ltest -lunwind

On arm64:

$ ./main
int throws lib
int caught main
$ ./main_unwind
int throws lib
int caught main

On amd64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
int throws lib
int caught main

We see that 1.2.1-9ubuntu0.1 from -proposed does not abort, and instead runs as expected. There is no change in behaviour on amd64. The package in -proposed fixes the problem, happy to mark as verified for Focal.

[Matthew Ruffell (mruffell)]

Performing verification for Bionic.

I started two instances on AWS, one c6g.medium (arm64) and a t2.micro (amd64).

I went through the reproducer listed in the testcase with libunwind-dev 1.2.1-8 from -release.

$ wget https://bugs.launchpad.net/ubuntu/+source/libunwind/+bug/1999104/+attachment/5635122/+files/libunwind.tar.gz
$ sudo apt install -y build-essential libunwind-dev
$ tar xvf libunwind.tar.gz && cd test
test/
test/lib.hpp
test/main.cpp
test/lib.cpp
test/Makefile
~/test$ make all
g++ -g -shared-libgcc -shared -fPIC -std=c++11 -o libtest.so lib.cpp
g++ -g -shared-libgcc -o main -L. -Wl,-rpath,. main.cpp -ltest
g++ -g -shared-libgcc -o main_unwind -L. -Wl,-rpath,. main.cpp -ltest -lunwind

On arm64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
terminate called after throwing an instance of 'int'
terminate called recursively
Aborted (core dumped)

On amd64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
int throws lib
int caught main

As expected, we see arm64 abort the execution of the reproducer.

I then installed 1.2.1-8ubuntu0.1 from -proposed and rebuilt the reproducers:

$ make clean
$ make all
g++ -g -shared-libgcc -shared -fPIC -std=c++11 -o libtest.so lib.cpp
g++ -g -shared-libgcc -o main -L. -Wl,-rpath,. main.cpp -ltest
g++ -g -shared-libgcc -o main_unwind -L. -Wl,-rpath,. main.cpp -ltest -lunwind

On arm64:

$ ./main
int throws lib
int caught main
$ ./main_unwind
int throws lib
int caught main

On amd64:

~/test$ ./main
int throws lib
int caught main
~/test$ ./main_unwind
int throws lib
int caught main

We see that 1.2.1-8ubuntu0.1 from -proposed does not abort, and instead runs as expected. There is no change in behaviour on amd64. The package in -proposed fixes the problem, happy to mark as verified for Bionic.

[William Ashley (washley)]

How long will it take for these changes to be released to Focal and Bionic?

[Matthew Ruffell (mruffell)]

Hi William,

I sincerely apologise for the delay.

Currently libunwind is stuck in -proposed due to benign autopkgtest regressions in the kernel packages.

If you go to the below page:

https://people.canonical.com/~ubuntu-archive/pending-sru.html

And search for "libunwind" you will see entries for Bionic and Focal.

It is SRU policy to not release a package with current autopkgtest regressions.

Now, I have spent more time than I am willing to admit on trying to debug these failures, and I have also asked the Kernel Team, several which took a look, and some Launchpad admins, and we are still a bit stuck. The problem does not reproduce locally, only on Launchpad builders.

For example, take the 4.15 Bionic Kernel:

https://autopkgtest.ubuntu.com/packages/l/linux/bionic/amd64

(it is a reverse dependency of libunwind, which is why it is selected for autopkgtest)

https://autopkgtest.ubuntu.com/results/autopkgtest-bionic/bionic/amd64/l/linux/20230110_115614_09e98@/log.gz

It rebuilds fine, but then runs into apt resolver trouble when running the kernel testsuite.

autopkgtest makes a dummy package, that contains the list of necessary dependencies to run the testsuite, dpkg -i to install the package, and then does an apt install -f to force dependency resolution. The dummy package is called autopkgtest-satdep.

https://paste.ubuntu.com/p/Cszfkvy47Z/

But it fails in strange ways, like not being able to select build-essential, even though it is already installed in the builder.

I am still trying to debug the root cause behind these autopkgtest regressions, which is why things have been delayed.

There is a provision in SRUs where they can be released as long as I can prove that the upload did not cause the regression:

https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

In which case, I may as well invoke this clause, since I don't wish to keep you waiting any longer.

I will try and get this package released within the week.

Thanks,
Matthew

[William Ashley (washley)]

Thank you for that thorough explanation and helping get this fix released, Matthew.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libunwind - 1.2.1-8ubuntu0.1

---------------
libunwind (1.2.1-8ubuntu0.1) bionic; urgency=medium

  * Manually enable C++ exception support only on i386 and amd64,
    it is known broken on several other architectures.
    Thanks to Bernhard Übelacker. (Closes: #923962) (LP: #1999104)

 -- Matthew Ruffell <email address hidden> Tue, 13 Dec 2022 12:12:23 +1300

[Chris Halse Rogers (raof)]

The verification of the Stable Release Update for libunwind has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package libunwind - 1.2.1-9ubuntu0.1

---------------
libunwind (1.2.1-9ubuntu0.1) focal; urgency=medium

  * Manually enable C++ exception support only on i386 and amd64,
    it is known broken on several other architectures.
    Thanks to Bernhard Übelacker. (Closes: #923962) (LP: #1999104)

 -- Matthew Ruffell <email address hidden> Tue, 13 Dec 2022 12:07:17 +1300

[Matthew Ruffell (mruffell)]

Hi William.

The libunwind SRU for Bionic and Focal have now been released to -updates. Their versions are 1.2.1-8ubuntu0.1 for Bionic, and 1.2.1-9ubuntu0.1 for Focal.

I just want to apologise for the significant delay in getting libunwind released. It really was a exceptional amount of time, and I'm sorry it took so long.

Since I wrote to you last, I root caused the issue and worked with Paride to resolve the regression that was introduced into autopkgtest itself.

The bug in autopkgtest was quite obscure, and it required the following to occur:

1. an all-proposed build (--apt-pocket=proposed with no package pinning)
2. multiple tests defined in d/t/control
3. the tests do not allow reusing the same testbed system

All these conditions were present in the kernel autopkgtests, and the result was that the change to allow apt pinning for -proposed caused _create_apt_pinning_for_packages() to be called incorrectly and it set a pinning for the -release pocket at 990, over -updates and -proposed, at 500 each, which meant that -release was being favoured over -proposed, and it caused all sorts of apt resolve issues.

The issue was introduced in:

commit 1c018c78de9d9421c0c358c900a37e545334cc66
From: Paride Legovini <email address hidden>
Date: Thu, 15 Dec 2022 21:47:02 +0100
Subject: Pin pockets with Pin-Priority: 500
Link: https://salsa.debian.org/ci-team/autopkgtest/-/commit/1c018c78de9d9421c0c358c900a37e545334cc66

The full explanation of the autopkgtest issues can be found in the below emails:

From myself to Paride
https://paste.ubuntu.com/p/44yFTBNBHh/

From Paride to myself:
https://paste.ubuntu.com/p/jtt5wh6BB2/

Paride's merge request;
https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/218

Final fix commit:
https://salsa.debian.org/ci-team/autopkgtest/-/commit/94b9bb8db3051123d7b29a7880420340a76c7b7e

The fix is in place on the Launchpad build infrastructure, and we re-ran all autopkgtests around libunwind and its reverse dependencies, and they all passed, leading us clear to release libunwind to -updates.

Again, I sincerely apologise for keeping you waiting for so long, and I thank you for your patience and understanding while I debugged autopkgtest.

Thanks,
Matthew

[William Ashley (washley)]

Thanks! I verified bionic and focal (via docker images) now successfully run the libunwind-using tests updated earlier.