update-initramfs generating unbootable initrd files when using stderred LD_PRELOAD hook

Bug #1729836 reported by Jeff Turner
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
initramfs-tools (Ubuntu)
Triaged
Low
Unassigned

Bug Description

After running 'update-initramfs -u -k $(uname -r)', Linux will not boot. With 'quiet splash' set in grub, it appears to freeze after the 'Loading ramdisk...' message, and if booted in recovery mode, I see the following kernel panic:

Freeing unused kernel memory: 948k
Freeing unused kernel memory: 104K
x86/mm: Chcked W+X mappings: passed, no W+X pages found.
Failed to execute /init (error -2)
Kernel panic - not syncing: No working init found. Try passing init= optin to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
CPU: 6 PID: 1 Comm: swapper/0 Not tainted 4.13.0-16-generic #19-Ubuntu
Hardware name: Dell Inc. OptiPlex 990/0D6H9T, BIOS A07 09/10/2011
Call Trace:
  dump_stack+0x63/0x8b
  ? rest_init+0xb0/0xc0
  panic+0xe4/0x23d
  ? putname+0x4b/0x50
  ? rest_init+0xc0/0xc0
  kernel_init+0xeb/0xfc
  ret_from_fork0x25/0x30
Kernel Offset: 0x1fc00000 from 0xffffffff8a000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
---[ end Kernel panic - no syncing: No working init found. Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.

(full screenshot at https://photos.app.goo.gl/XLNKOM3cDNHsfyFt2).

Examining the initrd files, here is a good one, as installed by the Ubuntu 17.10 installer:

# file /boot/initrd.img-4.13.0-16-generic
/boot/initrd.img-4.13.0-16-generic: ASCII cpio archive (SVR4 with no CRC)
# ls -la /boot/initrd.img-4.13.0-16-generic
-rw-r--r-- 1 root root 50764087 Nov 3 21:02 /boot/initrd.img-4.13.0-16-generic

and here is the file after 'update-initramfs -u -k 4.13.0-16-generic':

# file /boot/initrd.img-4.13.0-16-generic
/boot/initrd.img-4.13.0-16-generic: gzip compressed data, last modified: Thu Nov 2 10:51:42 2017, from Unix
root@jturner-home:/boot# ls -la /boot/initrd.img-4.13.0-16-generic
-rw-r--r-- 1 root root 50895514 Nov 3 21:01 /boot/initrd.img-4.13.0-16-generic

Good and bad initrds can be downloaded from https://drive.google.com/drive/folders/1reMWj1TohrQ4K5EbBMJPhw4xXUdBi0K9?usp=sharing

After downloading initrd.img-4.13.0-16-generic-broken, the panic can be replicated in qemu with:

kvm -m 1G -kernel /boot/vmlinuz-4.13.0-16-generic -initrd /boot/initrd.img-4.13.0-16-generic-broken

(tip obtained from https://lists.debian.org/debian-kernel/2016/02/msg00323.html)

I've experienced this problem with Ubuntu 17.04 and 17.10 on two separate PCs (both Dell OptiPlex 990s). Ubuntu boots fine initially, until one day I update the kernel or something, and a .deb postinst script calls update-initramfs and trashes that kernel's initrd file. If I run 'update-initramfs -u -k all', all my old initrd files are toast.

The "good" initrd format "ASCII cpio archive (SVR4 with no CRC)" is strange. I'm not getting many google hits on it. None of my other Ubuntu/Debian servers use that format in their initrds.

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: initramfs-tools 0.125ubuntu12
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu3.1
Architecture: amd64
Date: Fri Nov 3 21:33:12 2017
InstallationDate: Installed on 2017-08-05 (90 days ago)
InstallationMedia: Ubuntu 17.04 "Zesty Zapus" - Release amd64 (20170412)
PackageArchitecture: all
SourcePackage: initramfs-tools
UpgradeStatus: Upgraded to artful on 2017-10-24 (10 days ago)

Revision history for this message
Jeff Turner (jeffturner) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in initramfs-tools (Ubuntu):
status: New → Confirmed
Revision history for this message
Brian Murray (brian-murray) wrote :

Can your provide the terminal output of running update-initramfs with the -v switch? Thanks in advance.

Changed in initramfs-tools (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Jeff Turner (jeffturner) wrote :

Sure, I've attached update-initramfs -v output, and confirmed that the generated initrd is unusable, both with kvm and a real computer.

summary: - update-initramfs generating unbootable initrd files
+ update-initramfs generating unbootable initrd files when using stderred
+ LD_PRELOAD hook
Revision history for this message
Jeff Turner (jeffturner) wrote :

I think I figured out the cause. Basically PEBCAK..

There's a program called stderred (https://github.com/sickill/stderred) which colorizes stderr to differentiate it from stdout, via a LD_PRELOAD hook. I have had this set for years in my ~/.bashrc:

root@jturner-home:~# echo $LD_PRELOAD
/home/jturner/src/github/stderred/build/libstderred.so

When I examine the initrd file as follows, I see half the binaries in bin/ contain ANSI escape codes, and the libstderred.so library has been pulled in:

root@jturner-home:/boot# update-initramfs -u -k 4.13.0-25-generic
update-initramfs: Generating /boot/initrd.img-4.13.0-25-generic
root@jturner-home:/boot# mkdir bad
root@jturner-home:/boot# ( cd bad; zcat ../initrd.img-4.13.0-25-generic | cpio -idmv; )
root@jturner-home:/boot# ls bad/home/jturner/src/github/stderred/build/libstderred.so
bad/home/jturner/src/github/stderred/build/libstderred.so
root@jturner-home:/boot# ls -1 bad/bin
?[0m
?[0m?[1m?[31m[[?[0m?[1m?[31m
?[0m?[1m?[31macpid?[0m?[1m?[31m
?[0m?[1m?[31mash?[0m?[1m?[31m
?[0m?[1m?[31mawk?[0m?[1m?[31m
?[0m?[1m?[31mbasename?[0m?[1m?[31m
?[0m?[1m?[31mblockdev?[0m?[1m?[31m
?[0m?[1m?[31mcat?[0m?[1m?[31m
...
?[1m?[31m[?[0m?[1m?[31m
busybox
cat
chroot
cpio
dash
date
dd
...

The output of update-initramfs goes red halfway.

Perhaps update-initramfs could sanitize its environment, or explicitly unset LD_PRELOAD. Hard to be idiot-proof when there's such sophisticated idiots :)

Revision history for this message
Jeff Turner (jeffturner) wrote :
Changed in initramfs-tools (Ubuntu):
status: Incomplete → Triaged
importance: Undecided → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.