Revert back frame pointers for s390x (remove -fno-omit-frame-pointer but use -mbackchain)

[Frank Heimes]

SRU Justification:

[ Impact ]

 * The preferred way of doing stack unwinding on Linux on Z is via dwarf call frame information.
In absence of a dwarf unwinder (as in the Linux kernel) a stack chain can be maintained at runtime in addition to the dwarf unwinding information.

 * This allows for simple backtrace implementations, but imposes a small runtime overhead. For this to work, all code that might be part of backtrace must be built with the -mbackchain GCC option.

 * The -fno-omit-framepointer switch is neither necessary nor helpful in this context.
  Having a (soft/simulated) frame pointer does not improve backtraces at all on IBM Z.

 * However, forcing a frame pointer via the -fno-omit-frame-pointer option negatively affects performance for multiple reasons: extra prologue/epilogue overhead and fewer shrink-wrapping opportunities.

 * Given -fno-omit-frame-pointer does not provide any improvements (backtraces or otherwise) and only reduces performance, -fno-omit-frame-pointers should not be used on IBM Z.

 * So we are facing here a performance penalty without any gain - on this particular platform.

 * And sometimes (in rare cases like LP#2060108) frame pointers may even lead to failed builds.

[ Test Plan ]

 * Due to the above description of the impact and rationale,
   this pragmatic approach for testing is given:

 * Build the affected packages where frame-pointers should be reverted
   using the updated dpkg package (that incl. the modified build defaults)
   on (or for) this particular platform.

 * Now frame-pointer usage be checked in the following different ways:

 * 1) For the ease of use (and thanks to Julian Klode), there is this python
      test script available that allows to verify a binary in regard to
      frame pointers:
      https://gist.github.com/julian-klode/85e55553f85c410a1b856a93dce77208
      https://gist.githubusercontent.com/julian-klode/85e55553f85c410a1b856a93dce77208/raw/488b8509e6f23fe48f917961fe711b285dcb2e28/dwprod.py
      requires python3-pyelftools

 * 2) Another more manual way is to verify based on debug symbols like this:
      - find and install the ddeb package
      - maybe extract the file (e.g. unzstd)
      - use 'readelf -wi'
      - and grep for 'DW_AT_produce' (build options)
      - look for entries regarding frame-pointer
      The output may look similar to this:
      readelf -wi ./usr/lib/debug/lib/modules/6.8.0-38-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce
          <23> DW_AT_producer : (indirect string, offset: 0x7d): GNU AS 2.42
          <129> DW_AT_producer : (indirect string, offset: 0x3eef): GNU C11 13.2.0 -m64 -mpacked-stack -mbackchain -msoft-float -march=z13 -mtune=z16 -mindirect-branch=thunk-extern -mfunction-return=thunk-extern -mindirect-branch-table -mrecord-mcount -mnop-mcount -mfentry -mzarch -g -gdwarf-5 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fno-stack-protector -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fzero-call-used-regs=used-gpr -fno-inline-functions-called-once -falign-functions=8 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fsanitize=bounds-strict -fsanitize=shift -fsanitize=bool -fsanitize=enum -fPIC

 * 3) And maybe watching the build messages / log for the build options that
      were used (but that is probably not sufficient - it's better to inspect
      the output.)

[ Where problems could occur ]

 * The dpkg modifications could have been done erroneously.
   A dpkg test build and/or builds of other packages with the modified dpkg
   version in place would show this.

 * The settings in dpkg might be overwritten by other settings/packages.
   Tests like above, would show this.

 * One may think there could be issues in an environment where some packages
   have frame-pointer enabled and other don't.
   This is fine and was confirmed by IBM toolchain team and ours
   (as well as by a longer running <weeks> test system,
    with FP disabled in kernel, that showed no issues - like expected).

[ Other Info ]

 * These changes were implemented during the opening of the oracular series.
   The very same changes are backported to 24.04 LTS.

 * These only affect the ppc64el and s390x architectures,
   for other architectures it's a no-change upload.

 * We didn't see any fallout for these changes during the development
   on the oracular series, and therefore don't expect any fallout or
   regressions in 24.04 LTS either.

[Zixing Liu (liushuyu-011)]

Hi there,

The bug description only mentioned the situation with GCC. I wonder what the story is with other toolchains like Clang and/or other LLVM-based compilers (e.g. rustc/ldc/julia etc)? Some of the toolchains do support the `-mbacktrace`-like option on s390x, while the rest do not allow you to specify this option.

[Matthias Klose (doko)]

dpkg is fixed in oracular

[Łukasz Zemczak (sil2100)]

Frank, can you help me in writing up some test cases for both of these bugs? SO that we have formally a test case someone can follow or at least understand. Thanks!

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted dpkg into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.22.6ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted dpkg (1.22.6ubuntu6.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

aodh/unknown (s390x)
apache2/2.4.58-1ubuntu8.4 (armhf)
apache2/unknown (s390x)
apt/2.7.14build2 (s390x)
apt-cacher/unknown (s390x)
asymptote/unknown (armhf)
avifile/unknown (arm64, s390x)
bearssl/unknown (s390x)
bind-dyndb-ldap/unknown (s390x)
bind9/unknown (s390x)
black/unknown (armhf, s390x)
blhc/unknown (armhf)
check-pgbackrest/unknown (s390x)
chrony/4.5-1ubuntu4.1 (arm64)
cups/unknown (armhf)
devscripts/2.23.7 (arm64)
devscripts/unknown (s390x)
dgit/11.8 (arm64)
dh-elpa/unknown (s390x)
dh-make-perl/unknown (s390x)
ding/unknown (armhf)
extrepo/0.13 (armhf)
extrepo/unknown (s390x)
gobject-introspection/1.80.1-1 (arm64)
gobject-introspection/unknown (armhf)
gtk+2.0/2.24.33-4ubuntu1.1 (armhf)
gzip/unknown (s390x)
imagemagick/8:6.9.12.98+dfsg1-5.2build2 (arm64)
khronos-opencl-clhpp/unknown (s390x)
lapack/3.12.0-3build1 (arm64, armhf)
libcpucycles/unknown (s390x)
libgdata/0.18.1-6build2 (armhf)
libmceliece/unknown (armhf)
librandombytes/unknown (armhf, s390x)
libxmlada/24.0.0-2build2 (armhf)
llvm-toolchain-14/unknown (s390x)
llvm-toolchain-16/unknown (s390x)
llvm-toolchain-18/unknown (s390x)
mandos/unknown (armhf)
mosh/unknown (armhf, s390x)
needrestart/3.6-7ubuntu4.1 (arm64)
needrestart/unknown (s390x)
pkg-js-tools/unknown (s390x)
pkg-perl-tools/0.78 (armhf)
python-crcmod/unknown (s390x)
rdkit/unknown (armhf)
sgml-base/unknown (armhf)
snapd/2.63+24.04ubuntu0.1 (s390x)
software-properties/unknown (amd64, arm64, armhf, i386, ppc64el, s390x)
software-properties-qt/unknown (armhf)
sysbench/unknown (s390x)
uim/1:1.8.8-9.3ubuntu2 (armhf)
util-linux/2.39.3-9ubuntu6 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#dpkg

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Matthias Klose (doko)]

test rebuild of a package with this dpkg upload:
https://launchpad.net/~doko/+archive/ubuntu/ppa/+sourcepub/16312933/+listing-archive-extra

reference build:
https://launchpad.net/ubuntu/+source/lz4/1.9.4-1build1

- ppc64el: the -fno-omit-frame-pointer flags are gone
- s390x: only -mbackchain is used
- no build flag changes on other architectures

this addresses item 3) for the validation.

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted glibc into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.39-0ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Matthias Klose (doko)]

the failing autopkg tests are now down to two packages:

 - software-properties (all architectures)
   this is unrelated to the dpkg upload, it also fails for
   all other packages triggering this test in -proposed.

 - libxmlada (armhf)
   This is expected, needs updates for the 64bit time_t changes
   on armhf. I will follow-up on this with the gcc-13 and gcc-14
   uploads to 24.04 LTS.

The package itself looks ok, and also examining build logs and packages for the no-change uploads (see LP: #2076340) look ok.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.39-0ubuntu8.3) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

netcdf/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for dpkg has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.22.6ubuntu6.1

---------------
dpkg (1.22.6ubuntu6.1) noble-security; urgency=medium

  * SRU:
    - Disable framepointer on ppc64el. LP: #2064539.
    - Disable framepointer on s390x, leaving only -mbackchain. LP: #2064538.

 -- Matthias Klose <email address hidden> Wed, 17 Jul 2024 11:09:50 +0200

[Simon Chopin (schopin)]

I verified the fix for glibc:

root@glibc-fp-ibm:~# ./dwprod.py /usr/lib/s390x-linux-gnu/libc.so.6 | grep backchain | wc -l
1843
root@glibc-fp-ibm:~# ./dwprod.py /usr/lib/s390x-linux-gnu/libc.so.6 | grep no-omit-frame | wc -l
2
root@glibc-fp-ibm:~# dpkg -l libc6
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-===============-============-=================================
ii libc6:s390x 2.39-0ubuntu8.3 s390x GNU C Library: Shared libraries

(the two outliers for the no-omit-frame-pointer option are deliberate upstream decisions that have always been so, thus considered out of scope for this SRU)

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.39-0ubuntu8.3

---------------
glibc (2.39-0ubuntu8.3) noble; urgency=medium

  * Fix framepointer flags for s390x and ppc64el
    (LP: #2064538, LP: #2064539)

 -- Simon Chopin <email address hidden> Thu, 08 Aug 2024 16:47:42 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.40-1ubuntu1

---------------
glibc (2.40-1ubuntu1) oracular; urgency=medium

  * Merge 2.40-1 from Debian experimental (LP: #2073372)
    Dropped changes:
    - d/p/ubuntu/lv_LV-currency.diff: applied upstream
    - d/p/ubuntu/apparmor/0001-*: applied upstream

    Refreshed patches:
    - d/p/ubuntu/fix-fortify-source.patch: adjusted to take new code into
      account
  * debian/patches/git-updates.diff: update from upstream stable branch

 -- Simon Chopin <email address hidden> Fri, 26 Jul 2024 11:16:06 +0200

[Frank Heimes (fheimes)]

Did the verification for the kernel (linux-generic) like above and based on representative kernel modules (arch-specific and common):

noble:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
$ uname -a
Linux testlpar1 6.8.0-35-generic #35-Ubuntu SMP Mon May 20 15:36:54 UTC 2024 s390x s390x s390x GNU/Linux
$ sudo unzstd /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/arch/s390/crypto/aes_s390.ko.zst
/usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/arch/s390/crypto/aes_s390.ko.zst: 507329 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce | grep -c mbackchain
2
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0
$ sudo unzstd /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/fs/nfs/nfsv4.ko.zst
/usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/fs/nfs/nfsv4.ko.zst: 17383593 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c mbackchain
24
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-35-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0

oracular:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Oracular Oriole (development branch)
Release: 24.10
Codename: oracular
$ uname -a
Linux s1lp14 6.11.0-4-generic #4-Ubuntu SMP Tue Aug 20 14:03:40 UTC 2024 s390x s390x s390x GNU/Linux
$ sudo unzstd /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/s390/crypto/aes_s390.ko.zst
/usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/s390/crypto/aes_s390.ko.zst: 529441 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce | grep -c mbackchain
2
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0
$ sudo unzstd /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko.zst
[sudo] password for ubuntu:
/usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko.zst: 18990945 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c mbackchain
24
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0

Like one can see '-mbackchain' is set, but "-no-omit-frame-pointer" is not set, hence successful verified.

Closing the affected 'linux (Ubuntu)' as 'Fix Released' - and with that the project entry as well.

[bugproxy (bugproxy)]

------- Comment From <email address hidden> 2024-08-26 10:56 EDT-------
Fix was released to 24.04 (Noble). Therefore, we can close the bug.

Thanks everyone for your work!

==> Changing status to: "CLOSED"