Revert back frame pointers for ppc64el (remove -fno-omit-frame-pointer)

[Frank Heimes]

SRU Justification:

[ Impact ]

 * Power's Linux ABIs all require an explicit call chain be stored on the call stack frames which are all accessible via the stack pointer.

 * Therefore, having a (soft/simulated) frame pointer does not improve backtraces at all on Power.

 * However, forcing a frame pointer via the -fno-omit-frame-pointer option negatively affects performance for multiple reasons: extra prologue/epilogue overhead and fewer shrink-wrapping opportunities.

 * Given -fno-omit-frame-pointer does not provide any improvements (backtraces or otherwise) and only reduces performance, -fno-omit-frame-pointers should not be used on Power.

 * So we are facing here a performance penalty without any gain - on this particular platform.

 * And sometimes (in rare cases like LP#2060108) frame pointers may even lead to failed builds.

[ Test Plan ]

 * Due to the above description of the impact and rationale,
   this pragmatic approach for testing is given:

 * Build the affected packages where frame-pointers should be reverted
   using the updated dpkg package (that incl. the modified build defaults)
   on (or for) this particular platform.

 * Now frame-pointer usage be checked in the following different ways:

 * 1) For the ease of use (and thanks to Julian Klode), there is this python
      test script available that allows to verify a binary in regard to
      frame pointers:
      https://gist.github.com/julian-klode/85e55553f85c410a1b856a93dce77208
      https://gist.githubusercontent.com/julian-klode/85e55553f85c410a1b856a93dce77208/raw/488b8509e6f23fe48f917961fe711b285dcb2e28/dwprod.py
      requires python3-pyelftools

 * 2) Another more manual way is to verify based on debug symbols like this:
      - find and install the ddeb package
      - maybe extract the file (e.g. unzstd)
      - use 'readelf -wi'
      - and grep for 'DW_AT_produce' (build options)
      - look for entries regarding frame-pointer
      The output may look similar to this:
      readelf -wi ./usr/lib/debug/lib/modules/6.8.0-38-generic/kernel/arch/s390/crypto/aes_s390.ko | grep DW_AT_produce
          <23> DW_AT_producer : (indirect string, offset: 0x7d): GNU AS 2.42
          <129> DW_AT_producer : (indirect string, offset: 0x3eef): GNU C11 13.2.0 -m64 -mpacked-stack -mbackchain -msoft-float -march=z13 -mtune=z16 -mindirect-branch=thunk-extern -mfunction-return=thunk-extern -mindirect-branch-table -mrecord-mcount -mnop-mcount -mfentry -mzarch -g -gdwarf-5 -O2 -std=gnu11 -p -fshort-wchar -funsigned-char -fno-common -fno-strict-aliasing -fno-asynchronous-unwind-tables -fno-delete-null-pointer-checks -fno-allow-store-data-races -fno-stack-protector -ftrivial-auto-var-init=zero -fno-stack-clash-protection -fzero-call-used-regs=used-gpr -fno-inline-functions-called-once -falign-functions=8 -fstrict-flex-arrays=3 -fno-strict-overflow -fstack-check=no -fconserve-stack -fsanitize=bounds-strict -fsanitize=shift -fsanitize=bool -fsanitize=enum -fPIC

 * 3) And maybe watching the build messages / log for the build options that
      were used (but that is probably not sufficient - it's better to inspect
      the output.)

[ Where problems could occur ]

 * The dpkg modifications could have been done erroneously.
   A dpkg test build and/or builds of other packages with the modified dpkg
   version in place would show this.

 * The settings in dpkg might be overwritten by other settings/packages.
   Tests like above, would show this.

 * One may think there could be issues in an environment where some packages
   have frame-pointer enabled and other don't.
   This is fine and was confirmed by IBM toolchain team and ours
   (as well as by a longer running <weeks> test system,
    with FP disabled in kernel, that showed no issues - like expected).

[ Other Info ]

 * These changes were implemented during the opening of the oracular series.
   The very same changes are backported to 24.04 LTS.

 * These only affect the ppc64el and s390x architectures,
   for other architectures it's a no-change upload.

 * We didn't see any fallout for these changes during the development
   on the oracular series, and therefore don't expect any fallout or
   regressions in 24.04 LTS either.

[Matthias Klose (doko)]

dpkg is fixed in oracular

[Matthias Klose (doko)]

for convenience, the diff for the noble upload. we'll do a binary copy for the upload to -proposed, because the package needs to reach the security pocket as well.

[Frank Heimes (fheimes)]

Many thanks doko, the debdiff looks reasonable and good.
I did a test build (based on v1.22.6ubuntu6.1 https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/ppa/+sourcepub/16274696/+listing-archive-extra) and wasn't able to find an indication of FBs (in the debug symbols) anymore (using readelf -wi grep-ping for DW_AT_produce).
Same for LP#2064538.

[Łukasz Zemczak (sil2100)]

Frank, can you help me in writing up some test cases for both of these bugs? SO that we have formally a test case someone can follow or at least understand. Thanks!

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted dpkg into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dpkg/1.22.6ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Simon Chopin (schopin)]

Would running https://gist.githubusercontent.com/julian-klode/85e55553f85c410a1b856a93dce77208/raw/488b8509e6f23fe48f917961fe711b285dcb2e28/dwprod.py on /usr/lib/libc.so.6 to find the relevant flag (or absence thereof) be enough as a test case?

[Frank Heimes (fheimes)]

Hi @schopin, I agree - that is what is now under option 1) under test plan in the SRU Justification.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted dpkg (1.22.6ubuntu6.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

aodh/unknown (s390x)
apache2/2.4.58-1ubuntu8.4 (armhf)
apache2/unknown (s390x)
apt/2.7.14build2 (s390x)
apt-cacher/unknown (s390x)
asymptote/unknown (armhf)
avifile/unknown (arm64, s390x)
bearssl/unknown (s390x)
bind-dyndb-ldap/unknown (s390x)
bind9/unknown (s390x)
black/unknown (armhf, s390x)
blhc/unknown (armhf)
check-pgbackrest/unknown (s390x)
chrony/4.5-1ubuntu4.1 (arm64)
cups/unknown (armhf)
devscripts/2.23.7 (arm64)
devscripts/unknown (s390x)
dgit/11.8 (arm64)
dh-elpa/unknown (s390x)
dh-make-perl/unknown (s390x)
ding/unknown (armhf)
extrepo/0.13 (armhf)
extrepo/unknown (s390x)
gobject-introspection/1.80.1-1 (arm64)
gobject-introspection/unknown (armhf)
gtk+2.0/2.24.33-4ubuntu1.1 (armhf)
gzip/unknown (s390x)
imagemagick/8:6.9.12.98+dfsg1-5.2build2 (arm64)
khronos-opencl-clhpp/unknown (s390x)
lapack/3.12.0-3build1 (arm64, armhf)
libcpucycles/unknown (s390x)
libgdata/0.18.1-6build2 (armhf)
libmceliece/unknown (armhf)
librandombytes/unknown (armhf, s390x)
libxmlada/24.0.0-2build2 (armhf)
llvm-toolchain-14/unknown (s390x)
llvm-toolchain-16/unknown (s390x)
llvm-toolchain-18/unknown (s390x)
mandos/unknown (armhf)
mosh/unknown (armhf, s390x)
needrestart/3.6-7ubuntu4.1 (arm64)
needrestart/unknown (s390x)
pkg-js-tools/unknown (s390x)
pkg-perl-tools/0.78 (armhf)
python-crcmod/unknown (s390x)
rdkit/unknown (armhf)
sgml-base/unknown (armhf)
snapd/2.63+24.04ubuntu0.1 (s390x)
software-properties/unknown (amd64, arm64, armhf, i386, ppc64el, s390x)
software-properties-qt/unknown (armhf)
sysbench/unknown (s390x)
uim/1:1.8.8-9.3ubuntu2 (armhf)
util-linux/2.39.3-9ubuntu6 (arm64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#dpkg

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Matthias Klose (doko)]

test rebuild of a package with this dpkg upload:
https://launchpad.net/~doko/+archive/ubuntu/ppa/+sourcepub/16312933/+listing-archive-extra

reference build:
https://launchpad.net/ubuntu/+source/lz4/1.9.4-1build1

- ppc64el: the -fno-omit-frame-pointer flags are gone
- s390x: only -mbackchain is used
- no build flag changes on other architectures

this addresses item 3) for the validation.

[Łukasz Zemczak (sil2100)]

Hello Frank, or anyone else affected,

Accepted glibc into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/glibc/2.39-0ubuntu8.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Matthias Klose (doko)]

the failing autopkg tests are now down to two packages:

 - software-properties (all architectures)
   this is unrelated to the dpkg upload, it also fails for
   all other packages triggering this test in -proposed.

 - libxmlada (armhf)
   This is expected, needs updates for the 64bit time_t changes
   on armhf. I will follow-up on this with the gcc-13 and gcc-14
   uploads to 24.04 LTS.

The package itself looks ok, and also examining build logs and packages for the no-change uploads (see LP: #2076340) look ok.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted glibc (2.39-0ubuntu8.3) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

netcdf/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#glibc

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for dpkg has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dpkg - 1.22.6ubuntu6.1

---------------
dpkg (1.22.6ubuntu6.1) noble-security; urgency=medium

  * SRU:
    - Disable framepointer on ppc64el. LP: #2064539.
    - Disable framepointer on s390x, leaving only -mbackchain. LP: #2064538.

 -- Matthias Klose <email address hidden> Wed, 17 Jul 2024 11:09:50 +0200

[Simon Chopin (schopin)]

I verified the glibc SRU for Noble:

[before]
root@glibc-noble:~# dpkg -l libc6 | tail -n 1
ii libc6:ppc64el 2.39-0ubuntu8.2 ppc64el GNU C Library: Shared libraries
root@glibc-noble:~# ./dwprod.py /usr/lib/powerpc64le-linux-gnu/libc.so.6 | grep no-omit-frame-pointer | wc -l
1984

[after]
root@glibc-noble:~# dpkg -l libc6 | tail -n 1
ii libc6:ppc64el 2.39-0ubuntu8.3 ppc64el GNU C Library: Shared libraries
root@glibc-noble:~# ./dwprod.py /usr/lib/powerpc64le-linux-gnu/libc.so.6 | grep no-omit-frame-pointer | wc -l
2

The two remaining occurrences of -fno-omit-frame-pointer are from explicit upstream choices for specific object files (and have been enabled for years), which I deemed out of scope for this SRU.

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.39-0ubuntu8.3

---------------
glibc (2.39-0ubuntu8.3) noble; urgency=medium

  * Fix framepointer flags for s390x and ppc64el
    (LP: #2064538, LP: #2064539)

 -- Simon Chopin <email address hidden> Thu, 08 Aug 2024 16:47:42 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package glibc - 2.40-1ubuntu1

---------------
glibc (2.40-1ubuntu1) oracular; urgency=medium

  * Merge 2.40-1 from Debian experimental (LP: #2073372)
    Dropped changes:
    - d/p/ubuntu/lv_LV-currency.diff: applied upstream
    - d/p/ubuntu/apparmor/0001-*: applied upstream

    Refreshed patches:
    - d/p/ubuntu/fix-fortify-source.patch: adjusted to take new code into
      account
  * debian/patches/git-updates.diff: update from upstream stable branch

 -- Simon Chopin <email address hidden> Fri, 26 Jul 2024 11:16:06 +0200

[Frank Heimes (fheimes)]

Did the verification for the kernel (linux-generic) like above and based on representative kernel modules (arch-specific and common):

noble:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 24.04 LTS
Release: 24.04
Codename: noble
$ uname -a
Linux P10d-LPAR06 6.8.0-41-generic #41-Ubuntu SMP Fri Aug 2 21:00:36 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux
$ sudo unzstd /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko.zst
/usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko.zst: 338761 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko | grep DW_AT_produce | grep -c mbackchain
0
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0
$ sudo unzstd /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/fs//nfs/nfsv4.ko.zst
/usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/fs//nfs/nfsv4.ko.zst: 18009201 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/fs//nfs/nfsv4.ko | grep DW_AT_produce | grep -c mbackchain
0
$ readelf -wi /usr/lib/debug/lib/modules/6.8.0-41-generic/kernel/fs//nfs/nfsv4.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0

oracular:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu Oracular Oriole (development branch)
Release: 24.10
Codename: oracular
$ uname -a
Linux P10d-LPAR06 6.11.0-4-generic #4-Ubuntu SMP Tue Aug 20 14:54:18 UTC 2024 ppc64le ppc64le ppc64le GNU/Linux
$ sudo unzstd /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko.zst
/usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko.zst: 354129 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko | grep DW_AT_produce | grep -c mbackchain
0
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/arch/powerpc/crypto/aes-gcm-p10-crypto.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0
$ sudo unzstd /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko.zst
/usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko.zst: 19618441 bytes
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c mbackchain
0
$ readelf -wi /usr/lib/debug/lib/modules/6.11.0-4-generic/kernel/fs/nfs/nfsv4.ko | grep DW_AT_produce | grep -c no-omit-frame-pointer
0

(Btw. the negative test for mbackchain was done by intention, since this needs to be set for s390x, but for s390x only. The changes were done at the same time, verifying here, that is was really not set for ppc64el.)

Like one can see "-no-omit-frame-pointer" (and 'mbackchain') is not set, hence successful verified.

Closing the affected 'linux (Ubuntu)' as 'Fix Released' - and with that the project entry as well.