libc6-lse lets update-initramfs fail on AWS m6g instances

[Sorah Fukumori]

[Impact]

 * update-initramfs -u fails on arm64 m6g instances in AWS

[Test Case]

 * launch m6g instance in AWS
 * install libc6-lse (if not installed)
 * run $ update-initramfs -u
 * It should suceed
 * It should contain pthread, and libgcc_s libraries

[Regression Potential]

 * Adding one more path to libgcc_s1 resolution. This will still fail if something compiles libc6 for _two_ optimisations like /lib/$arch/foo/bar/libpthread.

[Other Info]

 * libphtread dlopens libgcc_s1, thus whenever libpthread is needed in the initrd libgcc_s1 must be copied in too. However the logic to find matching libgcc_s1 is broken for optimizied builds of libc6 without optimized build of libgcc_s1. I think libpthread should link against libgcc_s1 to prevent these issues.

 * Original bug report

With Ubuntu 20.04 on AWS m6g.* instance family, installing libc6-lse lets update-initramfs always fail with the following error:

ubuntu@ip-10-18-23-79:~$ sudo update-initramfs -u
update-initramfs: Generating /boot/initrd.img-5.4.0-1011-aws
E: /usr/share/initramfs-tools/hooks/btrfs failed with return 1.
update-initramfs: failed for /boot/initrd.img-5.4.0-1011-aws with 1.

## Steps to reproduce (on AWS)

### With focal 20200423 AMI

1. Find the following AMI and launch on m6g instance family

   ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20200423

2. Run: sudo apt update && sudo apt install libc6-lse
3. Try: sudo update-initramfs -u

### With focal 20200522 AMI

1. Find the following AMI and launch on m6g instance family

   ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20200522

2. Try: sudo update-initramfs -u

## Note

- The entire log of the above steps performed on 20200423 AMI is attached.
- Latest cloud-image AMI "ubuntu/images/hvm-ssd/ubuntu-focal-20.04-arm64-server-20200522" includes libc6-lse. On 20200522 AMI, this doesn't reproduce after removing libc6-lse manually.
- This doesn't reproduce on EC2 a1.* instance family.

## Expected behavior

Does not fail.

## Background to find this bug

As the 20200522 AMI includes libc6-lse out-of-the-box & apt-get upgrade pulls newer package that triggers update-initramfs, apt-get upgrade always fail on 20200522 AMI.

the following is an apport report on 20200423 AMI:

----

ProblemType: Bug
ApportVersion: 2.20.11-0ubuntu27
Architecture: arm64
CasperMD5CheckResult: skip
Date: Wed May 27 09:52:16 2020
Dependencies:
 gcc-10-base 10-20200411-0ubuntu1
 libc6 2.31-0ubuntu9
 libcrypt1 1:4.4.10-10ubuntu4
 libgcc-s1 10-20200411-0ubuntu1
 libidn2-0 2.2.0-2
 libunistring2 0.9.10-2
DistroRelease: Ubuntu 20.04
Ec2AMI: ami-061102f51d47b1c24
Ec2AMIManifest: (unknown)
Ec2AvailabilityZone: ap-northeast-1c
Ec2InstanceType: m6g.medium
Ec2Kernel: unavailable
Ec2Ramdisk: unavailable
Package: libc6-lse 2.31-0ubuntu9
PackageArchitecture: arm64
ProcCpuinfoMinimal:
 processor : 0
 BogoMIPS : 243.75
 Features : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
 CPU implementer : 0x41
 CPU architecture: 8
 CPU variant : 0x3
 CPU part : 0xd0c
 CPU revision : 1
ProcEnviron:
 LANG=C.UTF-8
 TERM=screen-256color
 PATH=(custom, no user)
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 5.4.0-1009.9-aws 5.4.30
SourcePackage: glibc
Tags: focal ec2-images
Uname: Linux 5.4.0-1009-aws aarch64
UpgradeStatus: No upgrade log present (probably fresh install)

[Sorah Fukumori (sorah)]

note that there's a difference of "atomics" in /proc/cpuinfo features between a1 instances and m6g instances.

[Dimitri John Ledkov (xnox)]

I'm failing to find a region with spot capacity for m6g instances. Which region did you launch your instance in such that I can try to reproduce and fix the issue?

[Dimitri John Ledkov (xnox)]

I managed to reproduce this.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.137ubuntu9

---------------
initramfs-tools (0.137ubuntu9) groovy; urgency=medium

  * Document add_mountroot_fail_hook correctly.

initramfs-tools (0.137ubuntu8) groovy; urgency=medium

  * Partially revert 0.133ubuntu12 upload:
    "* scripts/functions: Fix args in the add_mountroot_fail_check function."
    The arguments were correct, as used by lvm2 hook. LP: #1871154

initramfs-tools (0.137ubuntu7) groovy; urgency=medium

  * Fix copying libgcc_s when there is optimized pthreads installed.
    LP: #1880853

 -- Dimitri John Ledkov <email address hidden> Sat, 30 May 2020 16:33:08 +0100

[Brian Murray (brian-murray)]

Hello Sorah, or anyone else affected,

Accepted initramfs-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.136ubuntu6.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Sorah Fukumori (sorah)]

Confirmed the bug gets addressed with the initramfs-tools 0.136ubuntu6.1 on arm64 focal.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.136ubuntu6.1

---------------
initramfs-tools (0.136ubuntu6.1) focal; urgency=medium

  * SRU changes from Dimitri John Ledkov into focal:
    - Partially revert 0.133ubuntu12 upload:
      "* scripts/functions: Fix args in the add_mountroot_fail_check function."
      The arguments were correct, as used by lvm2 hook.
      LP: #1871154
    - Fix copying libgcc_s when there is optimized pthreads installed.
      LP: #1880853

 -- Gianfranco Costamagna <email address hidden> Mon, 01 Jun 2020 11:03:12 +0200

[Brian Murray (brian-murray)]

The verification of the Stable Release Update for initramfs-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Balint Reczey (rbalint)]

IMO this should go to -security as well because security updates can trigger it easily.

[Balint Reczey (rbalint)]

Also with the current fix - which is in itself is correct - only optimized libpthread libraries end up in initramfs which may prevent rebooting to systems without the same optimization.

I have opened LP: #1883883 to track that.

[Timo Aaltonen (tjaalton)]

Hello Sorah, or anyone else affected,

Accepted initramfs-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.130ubuntu3.10 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted initramfs-tools (0.130ubuntu3.10) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

zfs-linux/0.7.5-1ubuntu16.10 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dimitri John Ledkov (xnox)]

On a bionic system, with atomics & libc6-lse installed, I observed that whilst atomics libraries were present in the initrd, and libpthread. libgcc_s was not at all.

$ lsinitramfs /boot/initrd.img-5.3.0-1034-aws | grep atomics
lib/aarch64-linux-gnu/atomics
lib/aarch64-linux-gnu/atomics/libc.so.6
lib/aarch64-linux-gnu/atomics/libdl.so.2
lib/aarch64-linux-gnu/atomics/libc-2.27.so
lib/aarch64-linux-gnu/atomics/libm-2.27.so
lib/aarch64-linux-gnu/atomics/libpthread.so.0
lib/aarch64-linux-gnu/atomics/librt.so.1
lib/aarch64-linux-gnu/atomics/libpthread-2.27.so
lib/aarch64-linux-gnu/atomics/libdl-2.27.so
lib/aarch64-linux-gnu/atomics/librt-2.27.so
lib/aarch64-linux-gnu/atomics/libm.so.6

$ lsinitramfs /boot/initrd.img-5.3.0-1034-aws | grep libgcc
$

Installed initramfs-tools (,-core,-bin) version 0.130ubuntu3.10, and now libgcc_s is there

$ lsinitramfs /boot/initrd.img-5.3.0-1034-aws | grep libgcc
lib/aarch64-linux-gnu/libgcc_s.so.1

as needed by libpthread-2.27.so.

Verification done.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.130ubuntu3.10

---------------
initramfs-tools (0.130ubuntu3.10) bionic; urgency=medium

  * Cherrypick upstream commit to copy libgcc_s, as a dependency of
    libpthread including when there is optimized pthreads installed. LP:
    #1880853

 -- Dimitri John Ledkov <email address hidden> Wed, 02 Sep 2020 15:53:41 +0100