[MIR] lz4 by default

[Dimitri John Ledkov]

Use `lz4 -9 -l` compression for initramfs by default as discussed on ubuntu-devel.

This would also pull the lz4 package into main

https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040726.html

[Regression Potential]

We are trying to optimize for total boot speed, but performing a micro-optimization upon time to create/unpack kernel/initrd is an insufficient benchmark for total boot speed. This is because it ignores time to load the kernel/initrd, and whether the firmware/bootloader were able to stream decompress it whilst loading it. I.e. it is argued that in the real world, subsecond decompression gains are irrelevant if UEFI firmware, tftp boot, etc. take a lot longer than that to read extra 10s of MBs of boot material.

[TODO]
Measure pure i/o load speed with stopwatch, to figure out MB/s speed of loading initrds/kernel off FAT32, EXT4, TFTP, HTTP.
Re-evaluate if we should provide different compression mechanisms:
- ie. gzip instead of lz4 for most cases (revert)
- ie. xz for painful i/o cases (e.g. netboot)

I booted grub2 and measured loading largish amount of files, ie. $ date; initrd (hd0,gpt5)/initrd.img; initrd (hd0,gpt5)/initrd.img; initrd (hd0,gpt5)/initrd.img; initrd (hd0,gpt5)/initrd.img; initrd (hd0,gpt5)/initrd.img; date

To get a rough speed between 30 and 44 MB/s of loading these files off ext4 on nvme.

With lz4 initrd taking 67M, and gzip initrd taking 59M, the grub i/o penalty is 0.18s whilst I gain over a second in faster decompression time. Overall a win.

xz initrd is 36M meaning saving e.g. 0.8s of i/o time whilst gaining 2.4s of decompression time, meaning overall worse than gzip.

[Ubuntu Kernel Bot (ubuntu-kernel-bot)]

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1831736

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

[Launchpad Janitor (janitor)]

This bug was fixed in the package live-build - 3.0~a57-1ubuntu38

---------------
live-build (3.0~a57-1ubuntu38) eoan; urgency=medium

  * Stop setting LB_INITRAMFS_COMPRESSION default, and instead fallback to
    using initramfs-tools default. LB_INITRAMFS_COMPRESSION is now only to
    override whatever initramfs-tools' default compression is. This thus
    makes live-build default to lz4. LP: #1831736

 -- Dimitri John Ledkov <email address hidden> Wed, 05 Jun 2019 13:34:29 +0100

[Launchpad Janitor (janitor)]

This bug was fixed in the package livecd-rootfs - 2.592

---------------
livecd-rootfs (2.592) eoan; urgency=medium

  * Drop trying to mount removed maas squashfs.
  * Stop overriding initramfs compression default to lzma. LP: #1831736
  * Do not force lzma on ubuntu-core builds, the compress format default
    should be set universally inside initramfs-tools-ubuntu-core package
    instead of getting duplicated multiple times all over the place.

 -- Dimitri John Ledkov <email address hidden> Wed, 05 Jun 2019 13:55:06 +0100

[Steve Langasek (vorlon)]

Since these changes to the default initramfs compression are already being uploaded, it's critical that there be follow-through on the upgrader and on the sizing of /boot partitions to ensure that this is a smooth transition for our users.

https://lists.ubuntu.com/archives/ubuntu-devel/2018-March/040265.html
https://lists.ubuntu.com/archives/ubuntu-devel/2019-June/040729.html

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.133ubuntu4

---------------
initramfs-tools (0.133ubuntu4) eoan; urgency=medium

  * Switch default initramfs compression to lz4, faster than the current
    default gzip. LP: #1831736

 -- Dimitri John Ledkov <email address hidden> Wed, 05 Jun 2019 13:23:29 +0100

[Dimitri John Ledkov (xnox)]

Linux kernel compression was changed to lz4 with bugs
https://bugs.launchpad.net/bugs/1840934

[Dimitri John Ledkov (xnox)]

Sizing:

$ sudo du -sh /boot/* | grep -e grub -e 5.3.0-18
231K /boot/config-5.3.0-18-generic
8.0M /boot/grub
81M /boot/initrd.img-5.3.0-18-generic
4.5M /boot/System.map-5.3.0-18-generic
11M /boot/vmlinuz-5.3.0-18-generic

This is desktop system, amd64, with all microcodes, and linux-firmware, and signed grub.

For three kernels that brings my system to (11+81+4.5+0.2)*3+8 = 298.1 MB

Which is still within reasonable /boot sizing and need not to be bumped.

However, there are potential disk space saving that could be made:
 * on securebooted systems /boot/grub contains 8MB of modules that cannot be loaded at runtime
 * config is informational only, and is not strictly needed at boot
 * System.map is not strictly needed for boot
Which could save (4.5+0.2)*3+8=22.1 MB

Subiquity creates /boot at 1GB.
Ubiquity/partman-auto aims for (512 1024 768)
All are still reasonable for up to 10 kernel versions.

[Dimitri John Ledkov (xnox)]

However 298.1MB is larger than old installs.

[Steve Langasek (vorlon)]

On Tue, Oct 15, 2019 at 09:30:48AM -0000, Dimitri John Ledkov wrote:
> However 298.1MB is larger than old installs.

Do you mean that 298.1MB is larger than what old installs will guarantee for
a /boot partition? Which old releases in particular will be affected?
Shouldn't this be called out in the release notes?

For reference:

$ df -h /boot
Filesystem Size Used Avail Use% Mounted on
/dev/sda5 280M 116M 145M 45% /boot
$

With an Ubuntu 10.04.1 vintage install ;)