mkinitramfs fails with copy_file binary '/libgcc_s.so.[1-9]' not found

[Frank Heimes]

I recently did an 24.10 install and was surprised that the system hangs during the post-install reboot.

Investigation showed that this happens if the installer applies updates, which is done by default, if there is proper network connectivity and archive access.

In case of an install where I disabled the network, hence an offline install was done, the installation incl. post-install reboot was successful.

So I took such a successfully installed system (where no updates were applied) and noticed that currently the available updates are:
$ apt list --upgradable
iproute2/oracular-updates 6.10.0-2ubuntu1 s390x [upgradable from: 6.10.0-2]
linux-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
linux-headers-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
linux-image-generic/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
linux-libc-dev/oracular-updates 6.11.0-9.9 s390x [upgradable from: 6.11.0-8.8]
linux-tools-common/oracular-updates 6.11.0-9.9 all [upgradable from: 6.11.0-8.8]

So seemed to be kernel related, hence tried to update the kernel manually:
$ sudo apt install linux-image-generic/oracular-updates
Selected version '6.11.0-9.9' (Ubuntu:24.10/oracular-updates [s390x]) for 'linux-image-generic'
Upgrading:
  linux-generic linux-image-generic linux-tools-common
  linux-headers-generic linux-libc-dev

Installing dependencies:
  linux-headers-6.11.0-9 linux-modules-extra-6.11.0-9-generic
  linux-headers-6.11.0-9-generic linux-tools-6.11.0-9
  linux-image-6.11.0-9-generic linux-tools-6.11.0-9-generic
  linux-modules-6.11.0-9-generic

Suggested packages:
  fdutils linux-tools

Summary:
  Upgrading: 5, Installing: 7, Removing: 0, Not Upgrading: 1
  Download size: 65.3 MB
  Space needed: 165 MB / 21.2 GB available
  └─ in /boot: 43.1 MB / 1,876 MB available

Continue? [Y/n] Y
...
Setting up linux-generic (6.11.0-9.9) ...
Progress: [ 96%] [███████████████████████████████████████████████████████▋ ]
Progress: [ 96%] [███████████████████████████████████████████████████████▋ ]
Progress: [ 96%] [███████████████████████████████████████████████████████▋ ]
update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found█████▋ ]
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.
/etc/kernel/postinst.d/kdump-tools:
kdump-tools: Generating /var/lib/kdump/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
/etc/kernel/postinst.d/zz-zipl:
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.
Scanning processes...
Scanning processor microcode...
Scanning linux images...

Pending kernel upgrade!
Running kernel version:
  6.11.0-8-generic
Diagnostics:
  The currently running kernel version is not the expected kernel version
6.11.0-9-generic.

Restarting the system to load the new kernel will not be handled automatically,
so you should consider rebooting.

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor (qemu) binaries on this host.

I noticed these two lines, that have been spit out by update-initramfs:
update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found█████▋ ]
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found

Hence I tried to ran update-initramfs on the release kernel 6.11.0-8-8 and it happened there as well:

$ sudo update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-6.11.0-8-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.

So it's a initramfs-tools (mkinitramfs) issue rather than a kernel issue,
somewhere in copy_libgcc() (or before).

Seems to be a small issue, but is causing a big impact, since it leads to a broken default installation.

[Frank Heimes (fheimes)]

modified the hook-functions script (at line 252 ff) a bit to gather some more details and that's it:

$ grep -A 10 "copy_libgcc()" /usr/share/initramfs-tools/hook-functions
copy_libgcc() {
 local libdir library

 libdir="$1"
 echo "1" "$1"
 for library in "${libdir}"/libgcc_s.so.[1-9]; do
        echo "2" "${libdir}"
  copy_exec "${library}" || return
  echo "3" "${library}"
 done
}
$ sudo update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-6.11.0-8-generic
1 /lib/s390x-linux-gnu
2 /lib/s390x-linux-gnu
3 /lib/s390x-linux-gnu/libgcc_s.so.1
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
1
2
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
Using config file '/etc/zipl.conf'
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Run /lib/s390-tools/zipl_helper.device-mapper 252:1
Preparing boot device for LD-IPL: dm-0.
Done.
$ ls -l /lib/s390x-linux-gnu/libgcc_s.so.1
-rw-r--r-- 1 root root 80176 Sep 8 15:50 /lib/s390x-linux-gnu/libgcc_s.so.1
$

[Frank Heimes (fheimes)]

$ env --unset=LD_PRELOAD ldd /usr/lib/s390x-linux-gnu/libgcc_s.so.1
 linux-vdso64.so.1 (0x000003ffca8fe000)
 libc.so.6 => /lib/s390x-linux-gnu/libc.so.6 (0x000003ffabd00000)
 /lib/ld64.so.1 (0x000003ffac080000)
$

[Frank Heimes (fheimes)]

This is re-createable all time, a system is in place for further investigation (remote access) and testing.

[Frank Heimes (fheimes)]

The 'hook-functions' where the code is included is actually part of initramfs-tools-core:
dpkg -S /usr/share/initramfs-tools/hook-functions
initramfs-tools-core: /usr/share/initramfs-tools/hook-functions

[Benjamin Drung (bdrung)]

I see two things to further investigate:

1. copy_file: binary '/usr/lib/udev/dmsetup_env' not found

Who want to install this file?

2. copy_libgcc is called with the libdir parameter being empty

Can you instrument copy_exec to figure out who calls it that way?

[Benjamin Drung (bdrung)]

Might be unrelated, but this fix touches similar code: https://salsa.debian.org/kernel-team/initramfs-tools/-/commit/765db0268082dfaf0110441894abf3dafeff258b

[Frank Heimes (fheimes)]

Comparing it with a 24.04 (SAN/SCSI/multipath) installation, update-initramfs performs like this:

$ sudo update-initramfs -k all -u
update-initramfs: Generating /boot/initrd.img-6.8.0-47-generic
Using config file '/etc/zipl.conf'
Run /lib/s390-tools//zipl_helper.device-mapper /boot
Building bootmap in '/boot'
Adding IPL section 'ubuntu' (default)
Preparing boot device for LD-IPL: dm-0.
Done.

/usr/share/initramfs-tools/hooks/multipath is shipped with multipath-tools-boot
$ dpkg -S /usr/share/initramfs-tools/hooks/multipath
multipath-tools-boot: /usr/share/initramfs-tools/hooks/multipath

[Frank Heimes (fheimes)]

I have no idea what "/usr/lib/udev/dmsetup_env" is all about, since it is not in oracular, nor in noble, and also does not have a package it would belong to.

[Frank Heimes (fheimes)]

Line
copy_exec /usr/lib/udev/dmsetup_env
(as well as the one before: copy_exec /usr/sbin/dmsetup)
is in oracular's multipath-tools-boot v0.9.9-1ubuntu3 only, but not in noble's 0.9.4-5ubuntu8 -- the file /usr/share/initramfs-tools/hooks/multipath seem to differ quite a bit between the two package versions.

[Benjamin Drung (bdrung)]

Installing multipath-tools-boot let me reproduce the problem. The issue is clear, when looking at /usr/share/initramfs-tools/hooks/multipath: It calls copy_libgcc there without any parameters. So multipath-tools-boot is broken.

[Benjamin Drung (bdrung)]

```
$ schroot-wrapper -p initramfs-tools,multipath-tools-boot,linux-image-generic,zstd,vim -u root -c oracular
[...]
update-initramfs: Generating /boot/initrd.img-6.11.0-9-generic
mkinitramfs: copy_file: binary '/usr/lib/udev/dmsetup_env' not found
mkinitramfs: copy_file: binary '/libgcc_s.so.[1-9]' not found
```