raspi 64 bit Impish image fails update-initramfs with OOM on zstd

[Dan Bungert]

When attempting to 'apt upgrade' on the pi zero 2 with the 64 bit Impish image and nothing new installed to it, update-initramfs fails like so:

Setting up linux-firmware (1.201.1) ...
update-initramfs: Generating /boot/initrd.img-5.13.0-1008-raspi
Killed E: mkinitramfs failure zstd -q -19 -T0 137

A simple workaround for now is to switch to the old lz4 default:

$ sudo apt install lz4
$ sudo sed -i -e '/^COMPRESS=/ c COMPRESS=lz4' /etc/initramfs-tools/initramfs.conf

Original Workaround
===================

Adding a swap file to the setup allowed 'apt -f install' to run successfully.

How the device was setup
1) xzcat | dd the sdcard with the impish 64 bit server image
2) setup for serial over USB (dtoverlay=dwc2, modules-load=dwc2,g_serial, link <email address hidden>
) as suggested in https://www.tal.org/tutorials/raspberry-pi-zero-usb-serial-console
3) resize partition 2 - fdisk delete partition 2 and recreate to use the rest of the 16GB flash, e2fsck, resize2fs
4) cp bcm2710-rpi-3-b.dtb bcm2710-rpi-zero-2.dtb per https://waldorf.waveform.org.uk/2021/the-pi-zero-2.html
5) boot, connect to wireless, ssh in
6) apt update && apt upgrade
7) observe above issue

ProblemType: Package
DistroRelease: Ubuntu 21.10
Package: linux-firmware 1.201.1
ProcVersionSignature: Ubuntu 5.13.0-1009.10-raspi 5.13.14
Uname: Linux 5.13.0-1009-raspi aarch64
AlsaDevices:
 total 0
 crw-rw---- 1 root audio 116, 1 Nov 8 19:47 seq
 crw-rw---- 1 root audio 116, 33 Nov 8 19:47 timer
AlsaVersion: Advanced Linux Sound Architecture Driver Version k5.13.0-1009-raspi.
AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
ApportVersion: 2.20.11-0ubuntu71
Architecture: arm64
ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', '/dev/snd/timer'] failed with exit code 1:
CasperMD5CheckResult: unknown
Date: Mon Nov 8 21:00:59 2021
Dependencies:

ErrorMessage: installed linux-firmware package post-installation script subprocess returned error exit status 1
ImageMediaBuild: 20211013
IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
Lspci:

Lspci-vt: -[0000:00]-
Lsusb: Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Lsusb-t: /: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=dwc2/1p, 480M
PackageArchitecture: all
PciMultimedia:

ProcFB: 0 BCM2708 FB
ProcKernelCmdLine: coherent_pool=1M 8250.nr_uarts=1 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 bcm2708_fb.fbwidth=720 bcm2708_fb.fbheight=480 bcm2708_fb.fbswap=1 smsc95xx.macaddr=B8:27:EB:5C:DA:C8 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=LABEL=writable rootfstype=ext4 elevator=deadline rootwait fixrtc quiet splash modules-load=dwc2,g_serial
Python3Details: /usr/bin/python3.9, Python 3.9.7, python3-minimal, 3.9.4-1build1
PythonDetails: N/A
RelatedPackageVersions: grub-pc N/A
RfKill: Error: [Errno 2] No such file or directory: 'rfkill'
SourcePackage: linux-firmware
StagingDrivers: bcm2835_mmal_vchiq snd_bcm2835 bcm2835_isp bcm2835_v4l2 bcm2835_codec vc_sm_cma
Title: package linux-firmware 1.201.1 failed to install/upgrade: installed linux-firmware package post-installation script subprocess returned error exit status 1
UpgradeStatus: No upgrade log present (probably fresh install)
acpidump:

[Dave Jones (waveform)]

This is down to initramfs-tools switching to zstd by default, which has such huge memory requirements that it gets killed by OOM on the Zero 2 (and presumably the 3A+?) which only has 512MB of RAM. The default should be switched back to lz4, at least on the Pi images.

Note to self: add a test for update-initramfs -u to the ISO tracker for the Pi server images to ensure it gets tested across the platforms.

[Dave Jones (waveform)]

Adding ubuntu-settings as that's where the override for the raspi will be added

[Dave Jones (waveform)]

The attached debdiff fixes this issue in ubuntu-settings (and restores the 21.04.3 changes that were previously missing). Packages have been built and tested in the following PPA:

  https://launchpad.net/~waveform/+archive/ubuntu/ubuntu-settings

[Dave Jones (waveform)]

Some further research on this [1] suggests gzip may be a preferable choice on the smaller platforms (lower memory usage, and better compression anyway), plus this would alleviate the need for including lz4 in the images.

I need to do some further analysis of the decompression to see if it is still suitable there too.

[1] https://github.com/waveform80/compression/

[Dave Jones (waveform)]

Just to clarify, while gzip may be preferable this shouldn't prevent the lz4 fix from landing. It's a "good enough" workaround for a bug that prevents people from running apt-upgrade (which is far more important than "choice of compression algorithm").

[Sebastien Bacher (seb128)]

Upload to the current Ubuntu serie, Dave did you want to SRU that to impish? If so could you work on an update patch and SRUfy the reports?

[Launchpad Janitor (janitor)]

This bug was fixed in the package ubuntu-settings - 22.04.1

---------------
ubuntu-settings (22.04.1) jammy; urgency=medium

  * Reinstate changes from 21.04.3:
    - Adding 99-gpio.rules udev rules for Raspberry Pi GPIO functionality
      (LP: #1923672)
    - Replace netplan eth0 renaming rules for raspi with networkd link unit
      (LP: #1922266)
  * Override initramfs compression to lz4 to prevent OOM on 512MB platforms,
    and add a dependency on lz4 to d/control (LP: #1950214)
  * d/control: Correct spelling of raspberry

 -- Dave Jones <email address hidden> Wed, 15 Dec 2021 15:29:10 +0100

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.