Support compressed kernel modules in initramfs-tools and kernel

[Dimitri John Ledkov]

--- initramfs-tools

[Impact]

 * Initramfs-tools already supports compressed kernel modules. However, in focal and impish it does so inefficiently. It is always better to have a compressed initramfs of uncommpressed kernel modules, than a compressed initrd of compressed kernel modules. Thus when kernel module compression is turned on, it is prudent for initramfs-tools to pre-uncompress kernel modules when building initramfs.

[Test Plan]

 * Compress all kernel modules with xz for a current kernel, check that all of them have .ko.xz extension and no .ko ones available

 * Rerun depmod

 * Update initramfs with update-initramfs -u

 * lsinitramfs contents and check that all kernel modules are present, and are uncompressed (.ko extension)

[Where problems could occur]

 * This optimization for compressed kernel modules will make initramfs build time longer (due to decompression) whilst improving bootspeed (overall smaller size of the initrd).

[Other Info]

 * Original bug report re kernel feature

--- linux

Symbol: MODULE_COMPRESS_ZSTD [=n]
Type : bool

= Impacts to measure and observe =

== Disk space ==

* Inspect linux-modules-* and linux-modules-extra* deb package Installed-Size and Download-Size changes, i.e.

$ apt show linux-modules-5.8.0-53-generic linux-modules-extra-5.8.0-53-generic | grep -e Package: -e Size:

Package: linux-modules-5.8.0-53-generic
Installed-Size: 81.5 MB
Download-Size: 15.5 MB

Package: linux-modules-extra-5.8.0-53-generic
Installed-Size: 215 MB
Download-Size: 41.5 MB

In theory, there should not be a significant change in the Download-size. It is desired that there is a significant reduction in Installed-Size. Modules take up about 300MB and normally one has upto three kernel version installed, resulting in about of 1GB of disk space that one constantly pays for.

== Boot Speed ==

In theory, boot speed may either improve or regress. It depends if disk IO is slower than decompression speed, meaning loading compressed modules is faster.

Also one has to observe the changes in the initrd size. zstd(zstd) compression may result in slight growth, which shouldn't impact boot speed too much.

= Outcomes =

If installed size savings can be achieved without regressing bootspeed we should enable CONFIG_MODULE_COMPRESS_ZSTD=y by default.

[Colin Ian King (colin-king)]

Needed a bit of tweaking to get this benchmarked. Had to enable zstd in kmod package and also apply a hack to the kernel package building. (See attachment).

So, the installed compressed kernel modules are 27.1 MB compared to the uncompressed 99.1 MB. The compressed extra modules are 103 MB compared to the uncompressed modules at 296 MB.

linux-modules-5.13.0-8-generic installed size:
  zstd compressed modules: 27.1 MB
  normal uncompressed: 99.1 MB

linux-modules-extra-5.11.0-18-generic installed size:
  zstd compressed modules: 103.0 MB
  normal uncompressed: 206.0 MB

The downside is that compression for the deb and the initramfs does not do so well. It seems that compressing a lot of uncompressed modules is better than compressing already compressed modules:

linux-modules-5.13.0-8-generic deb size:
  zstd compressed modules: 19.9 MB
  normal uncompressed: 17.5 MB

linux-modules-extra-5.11.0-18-generic deb size:
  zsd compressed modules: 95.08 MB
  normal uncompressed: 48.70 MB (I double checked the contents of the deb)

The initramfs size:
  zsd compressed modules: 49.77 MB
  normal uncompressed: 41.24 MB

For boot times, the kernel + zstd modules for ZSD and uncompressed kernel modules is almost the same, zstd is fractionally faster but this is within the margins of measured error.

*HOWEVER*, the zstd initramfs does not compress already compressed zstd kernel modules well, so the initramfs size is actually larger, so this impacts on the time to load the initramfs, so we end up with slower boots with zstd module compression.

Attached is the data in a libre office spread sheet

[Colin Ian King (colin-king)]

Attached is the diff of the kernel to build zstd modules. This is a hack for the moment just to get benchmarks.

[Dimitri John Ledkov (xnox)]

I like smaller install size.
In many cases we boot without initrd by default (gcp, azure, aws, kvm).
Does it then make sense to enable this for the cloud kernels?

I ponder, if i can hack initramfs-tools to append compressed kernel modules as an uncompressed cpio archive to the initramfs, instead of having it together with the userspace bits all compressed together.
Separately, are the modules signed, then compressed, or compressed then signed?
Cause the other alternative is to decompress the modules whilst including in the initramfs. That way we should get unchanged initrd sizes.

So I guess I need to hack initramfs-tools more.

[Colin Ian King (colin-king)]

Note that the download size is much larger. Doesn't that impact on the amount of data that users download (which is quite large considering the number of kernel updates we do)?

[Colin Ian King (colin-king)]

Note also the boot speed is not much different, so the I doubt we see any noticeable boot performance wins on cloud images.

[Dimitri John Ledkov (xnox)]

Whilst download size is larger, it is at no additional cost to our end users, especially in the cloud. Majority of our clouds offer free transfers inside the availability zone for which mirrors are provided. Obviously it will increase the disk/cache storage size requirements of our mirrors.

Our end users have about 3 sets of kernels installed on disk, for which they do pay more per GB between 2 to 12 cents (depending on technology IOPS / hdd / sda / nvme). All things being equal our users would choose larger download sizes to gain smaller install size.

I assume that we have vastly larger install base, compared to number of our mirrors.

I also hope that our install base is not larger than the total bandwidth available of our mirrors, such that this switch would cause collapse of our mirror network.

I also notice that kernel uses default options for zstd module compression, instead of using -19 like it recommends for the initrds.
I also notice that kmod in focal doesn't support zstd.
Imho we should fix above before turning this on.

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.140ubuntu8

---------------
initramfs-tools (0.140ubuntu8) jammy; urgency=medium

  * Cherry-pick merge request !26 to support xz compressed
    firmware. Uncompressed firmware is preffered, with a fallback to use
    compressed firmware files. LP: #1942260
  * Cherry-pick merge request !51 to decompress compressed kernel modules
    for boot speed performance. LP: #1932329
  * Cherry-pick partially merge request !50 to fix ftbfs with new
    shellcheck.
  * Fix shellcheck issues in ubuntu patches.

 -- Dimitri John Ledkov <email address hidden> Tue, 19 Oct 2021 11:49:38 +0100

[Brian Murray (brian-murray)]

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

[Dimitri John Ledkov (xnox)]

moving linux tasks to https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2028568