[binutils] Prevent GOT access rewrite for certain symbols

Bug #1903814 reported by bugproxy
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Fix Released
High
Skipper Bug Screeners
binutils (Debian)
New
Unknown
binutils (Ubuntu)
Fix Released
Undecided
Unassigned
Bionic
Fix Released
Undecided
Canonical Foundations Team

Bug Description

[Impact]

 * In s390 kernel context, this bug manifests itself as random errors and infinite loops.

[Test Case]

 * Needs to be confirmed by IBM
 * Build-time tests-suite (applied upstream) backported to Bionic in
   + ld/testsuite/ld-s390/gotreloc-1.s
   + ld/testsuite/ld-s390/gotreloc-1.ver
   + ld/testsuite/ld-s390/gotreloc_31-1.dd
   + ld/testuite/ld-s390/gotreloc_64-1.dd
 * If you build the kernel with CONFIG_DEBUG_INFO_BTF, there is a 50% chance that you will see "Failed verification: in-kernel BTF is malformed" during boot

[Where problems could occur]

 * Binutils is a base toolchain package
   - A problem could potentially affect the whole system
   - With compiler/linker errors
   - Or random errors in the produced binaries
 * This patch touches only architecture specific code in bfd/elf64-s390.c
   - It would only affect the s390x architecture in this case

[Other Info]

 * While testing the fix in Bileto, we found one pending autopkgtest regression with linux/amd64 4.15.0-135.139 which is resolved in 4.15.0-136.140 (currently in -proposed).
 * The failed test in snapcraft is not a regression, as it never passed before.

== Original Description ==
Please backport the following bugfix into Ubuntu LTS: https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=e6213e09ed0e

Some relevant historic links:
Debian bugreport: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961736
glibc bugreport: https://sourceware.org/bugzilla/show_bug.cgi?id=18960

In s390 kernel context, this bug manifests itself as random errors and infinite loops, so it's fairly severe.

These are the current versions of binutils:
Package binutils

    xenial (16.04LTS) (devel): GNU assembler, linker and binary utilities
    2.26.1-1ubuntu1~16.04.8 [security]: amd64 i386
    2.26-8ubuntu2 [ports]: arm64 armhf powerpc ppc64el s390x
    xenial-updates (devel): GNU assembler, linker and binary utilities
    2.26.1-1ubuntu1~16.04.8: amd64 arm64 armhf i386 powerpc ppc64el s390x
    bionic (18.04LTS) (devel): GNU assembler, linker and binary utilities
    2.30-21ubuntu1~18.04.4 [security]: amd64 i386
    2.30-15ubuntu1 [ports]: arm64 armhf ppc64el s390x
    bionic-updates (devel): GNU assembler, linker and binary utilities
    2.30-21ubuntu1~18.04.4: amd64 arm64 armhf i386 ppc64el s390x
    focal (20.04LTS) (devel): GNU assembler, linker and binary utilities
    2.34-6ubuntu1: amd64 arm64 armhf i386 ppc64el s390x
    groovy (devel): GNU assembler, linker and binary utilities
    2.35.1-1ubuntu1: amd64 arm64 armhf i386 ppc64el s390x

The patch applies fine to 2.26 and 2.30 (except for tests, but we don't need them). We don't need it on 2.32+.

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-189170 severity-high targetmilestone-inin2104
Changed in ubuntu:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
affects: ubuntu → binutils-s390x-cross (Ubuntu)
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2020-11-11 03:58 EDT-------
Please provide the patch for bionic. We don't see an urgent need for Xenial(runs out of base support in April 2021).

Revision history for this message
Frank Heimes (fheimes) wrote :

According to the affected binutils versions and the communication with IBM it's sufficient to get this fixed in Bionic.

no longer affects: binutils-s390x-cross (Ubuntu)
Changed in ubuntu-z-systems:
importance: Undecided → High
Changed in binutils (Ubuntu):
status: New → Fix Released
Changed in binutils (Ubuntu Bionic):
assignee: nobody → Canonical Foundations Team (canonical-foundations)
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → Skipper Bug Screeners (skipper-screen-team)
status: New → Triaged
tags: added: fr-920
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

"The patch applies fine to 2.26 and 2.30 (except for tests, but we don't need them)."

but we do. normally whenever doing toolchain fixes and updates, unittests are expected to be backported as all of them are exercised before allowing to release the update.

Changed in binutils (Debian):
status: Unknown → New
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2020-11-18 08:42 EDT-------
Ok, let me see if I can port tests as well.

Revision history for this message
bugproxy (bugproxy) wrote : for 18.04

------- Comment on attachment From <email address hidden> 2020-11-23 11:20 EDT-------

The attached patch applies cleanly to binutils-2_30 tag and binutils_2.30-21ubuntu1~18.04.4ubuntu1.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-01-25 07:11 EDT-------
Hello, Ubuntu! Did you have a chance to evaluate the attached 18.04 patch?

Changed in binutils (Ubuntu Bionic):
status: New → In Progress
status: In Progress → New
Revision history for this message
Lukas Märdian (slyon) wrote :

I can confirm that the patch applies and the build seems to pass OK in a -security enabled PPA (https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4439).

But it produces some test failures/regressions, which we need to work on before we can get this moving: https://bileto.ubuntu.com/excuses/4439/bionic.html

Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-02-15 09:35 EDT-------
To me they all look like intermittent problems (timeouts on arm, 404s when installing packages on Intel). Is it possible to simply retrigger those?

Revision history for this message
Lukas Märdian (slyon) wrote :

Yes, the regressions seem to be unrelated to this change. Some can be resolved by retriggering. Others might need an updated kernel package (e.g. the 404 errors seem to be fixed in linux 4.15.0-136.140)

I've updated the patch headers a bit and re-uploaded the package using a proper version number, which we can later copy into the archive.

Revision history for this message
Lukas Märdian (slyon) wrote :

So the regression tests are looking good so far. I'm currently still verifying that the regression with the linux kernel on Intel (404s) is fixed with 4.15.0-136.140 (from -proposed). If that works out, I'll copy the packages to the archive for verification by the SRU team.

I've SRU'ifyed the description of this bug report. Is there any more specific test case you could give to verify that this bug is indeed fixed? How to reproduce those "random errors and infinite loops"?

description: updated
Revision history for this message
bugproxy (bugproxy) wrote :

------- Comment From <email address hidden> 2021-02-17 11:51 EDT-------
Here is the original complaint: https://lore.kernel.org/bpf/20200611205040.GA1853644@krava/

If you build the kernel with CONFIG_DEBUG_INFO_BTF, there is a 50% chance that you will see "Failed verification: in-kernel BTF is malformed" during boot. There was also a hang, reported by one of our kernel team members, that we tracked down to the same issue, but unfortunately I cannot find any details about it right now.

Lukas Märdian (slyon)
description: updated
Lukas Märdian (slyon)
description: updated
Lukas Märdian (slyon)
Changed in binutils (Ubuntu Bionic):
status: New → In Progress
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Triaged → In Progress
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Please test proposed package

Hello bugproxy, or anyone else affected,

Accepted binutils into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/binutils/2.30-21ubuntu1~18.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in binutils (Ubuntu Bionic):
status: In Progress → Fix Committed
tags: added: verification-needed verification-needed-bionic
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: In Progress → Fix Committed
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (binutils/2.30-21ubuntu1~18.04.5)

All autopkgtests for the newly accepted binutils (2.30-21ubuntu1~18.04.5) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

apport/2.20.9-0ubuntu7.23 (amd64)
glibc/2.27-3ubuntu1.4 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#binutils

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Lukas Märdian (slyon) wrote :

Both autopkgtest regressions have turned out to be intermittent failures and have been resolved by retriggering the corresponding tests.

Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-02-25 07:36 EDT-------
binutils-s390x-linux-gnu s390x 2.30-21ubuntu1~18.04.5 works fine.

Tested as follows:

#include <stdio.h>
char a __attribute((aligned(1)));
char b __attribute((aligned(1)));
int main() { printf("%p %p\n", &a, &b); }

prints the following:

0x2aa00c02014 0x2aa00c02014

on the latest bionic (the pointers are the same - bad).

After the upgrade to bionic-proposed it prints:

0x2aa2ed82015 0x2aa2ed82014

(the pointers are different - good).

Revision history for this message
Frank Heimes (fheimes) wrote :

Ok thx Ilyia, with that I'm going to update the tags accordingly ...

tags: added: verification-done verification-done-bionic
removed: verification-needed verification-needed-bionic
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package binutils - 2.30-21ubuntu1~18.04.5

---------------
binutils (2.30-21ubuntu1~18.04.5) bionic; urgency=medium

  * d/p/lp1903814-lp1903814-s390_prevent_GOT_access.patch:
    Prevent GOT access rewrite for certain symbols (LP: #1903814)
    Thanks to Ilya Leoshkevich for porting the upstream patch to v2.30:
    https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=e6213e09ed0ef6ee40d00981c38baac6964aa95b

 -- Lukas Märdian <email address hidden> Fri, 12 Feb 2021 09:45:35 +0100

Changed in binutils (Ubuntu Bionic):
status: Fix Committed → Fix Released
Revision history for this message
Łukasz Zemczak (sil2100) wrote : Update Released

The verification of the Stable Release Update for binutils has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
status: Fix Committed → Fix Released
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-03-01 06:32 EDT-------
IBM Bugzilla status->closed, Fix Release with all requested distros

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Bug attachments

Remote bug watches

Bug watches keep track of this bug in other bug trackers.