initramfs-tools & kernel: use zstd as the default compression method

Bug #1931725 reported by Colin Ian King
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu on IBM z Systems
Undecided
bugproxy
initramfs-tools (Ubuntu)
Undecided
Unassigned
linux (Ubuntu)
Undecided
Seth Forshee
linux-oracle (Ubuntu)
Undecided
Unassigned

Bug Description

Turns out that loading is always the slow part in loading initramfs into memory and decompressing it since decompression is always the final 10-20% or so of the task. It therefore makes sense to use a good compressor that shrinks the initramfs as much as possible with little decompression overhead.

Benchmarking zstd vs lz4 shows that while zstd can be ~5x slower in decompression, the image size is much smaller with zstd than lz4, and since ~80-90% of the boot time is loading the image it makes sense to use zstd.

Attached is a libreoffice spread sheet showing typical load and decompression times for a fairly standard 3.4 GHZ intel box with data for load times for a 5400 RPM, 7200 RPM and SATA SSD drives.

The conclusions from the test results (attached) show:

1. Loading time is always significantly slower than decompression time.
2. ZSTD is 5x slower than LZ4 in decompression speed but produces far
better compressed images
3. Given that loading time is the major factor in loading +
decompression, ZSTD is best for kernel and initramfs boot timings.

(Also refer to https://kernel.ubuntu.com/~cking/boot-speed-eoan-5.3/kernel-compression-method.txt for some raw data on drive load speeds for the same UEFI box I did a couple of years ago).

amd64 supports zstd, but s390x does not. Will use this bug to enable zstd support on s390x.

Upstream submitted patch https://lore<email address hidden>/T/#u

Revision history for this message
Colin Ian King (colin-king) wrote :
description: updated
description: updated
Changed in linux (Ubuntu):
assignee: nobody → Seth Forshee (sforshee)
Changed in initramfs-tools (Ubuntu):
status: New → Fix Committed
Revision history for this message
Colin Ian King (colin-king) wrote :

Thanks Dimitri!

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package initramfs-tools - 0.140ubuntu5

---------------
initramfs-tools (0.140ubuntu5) impish; urgency=medium

  * Switch default initramfs compression to zstd, as it produces the lower
    overall boot time. LP: #1931725

 -- Dimitri John Ledkov <email address hidden> Fri, 11 Jun 2021 16:55:30 +0100

Changed in initramfs-tools (Ubuntu):
status: Fix Committed → Fix Released
summary: - initramfs-tools: use zstd as the default compression method
+ initramfs-tools & kernel: use zstd as the default compression method
description: updated
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

@ IBM can you please review the upstream patch and merge it into the the s390 tree ? https://lore<email address hidden>/T/#u

description: updated
Frank Heimes (fheimes)
Changed in ubuntu-z-systems:
assignee: nobody → bugproxy (bugproxy)
tags: added: reverse-proxy-bugzilla s390x
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Vasily Gorbik is reviewing this patch.

bugproxy (bugproxy)
tags: added: architecture-s39064 bugnameltc-193300 severity-high targetmilestone-inin2004
Revision history for this message
bugproxy (bugproxy) wrote : Comment bridged from LTC Bugzilla

------- Comment From <email address hidden> 2021-06-16 07:19 EDT-------
FWIW, with Z15 and later we do have a zlib HW accelerator on the chip. So zlib would also be an option - assuming that z15 will be the most common Z pretty soon.
From my checks the HW decompression this was really fast.
How does zlib compression ratio (maybe try level 6 and 9) compare here?

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

decompression speed only needs to be faster than i/o speed, once that is reached the best compression ratio results in the fastest bootspped.

for kernel image zstd is used with -22 --ultra, thus I can compare it with zlib -9.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

@cborntra

v5.13 ubuntu kernel's s390 configuration with zstd -22 --ultra compression is 8.5 MB, whereas gzip -9 is 11M.

Thus for gzip to win at bootspeed the decompression speed has to compensate for 2.5M of i/o and be faster than zstd.

Unaccelerated decompression comparison still gives me faster decompression with less i/o with zstd compressed kernel image.

At Canonical we still do not have z15 mainframe available for us to benchmark this.

Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote :

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

tags: added: verification-needed-focal
Revision history for this message
Ubuntu Kernel Bot (ubuntu-kernel-bot) wrote : Missing required logs.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1931725

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Changed in linux (Ubuntu):
status: New → Incomplete
Frank Heimes (fheimes)
tags: added: bot-stop-nagging
Changed in ubuntu-z-systems:
status: New → Fix Released
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux - 5.13.0-14.14

---------------
linux (5.13.0-14.14) impish; urgency=medium

  * impish/linux: 5.13.0-14.14 -proposed tracker (LP: #1938565)

  * Miscellaneous Ubuntu changes
    - SAUCE: Revert "UBUNTU: SAUCE: random: Make getrandom() ready earlier"
    - SAUCE: random: properly make getrandom() ready earlier

  * Miscellaneous upstream changes
    - seq_buf: Fix overflow in seq_buf_putmem_hex()
    - bpf: Fix integer overflow in argument calculation for bpf_map_area_alloc
    - ext4: cleanup in-core orphan list if ext4_truncate() failed to get a
      transaction handle
    - ext4: fix kernel infoleak via ext4_extent_header
    - ext4: fix overflow in ext4_iomap_alloc()
    - ext4: return error code when ext4_fill_flex_info() fails
    - ext4: correct the cache_nr in tracepoint ext4_es_shrink_exit
    - ext4: remove check for zero nr_to_scan in ext4_es_scan()
    - ext4: fix avefreec in find_group_orlov
    - ext4: use ext4_grp_locked_error in mb_find_extent

 -- Andrea Righi <email address hidden> Mon, 02 Aug 2021 14:23:08 +0200

Changed in linux (Ubuntu):
status: Incomplete → Fix Released
Changed in linux-oracle (Ubuntu):
status: New → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package linux-oracle - 5.13.0-1008.10

---------------
linux-oracle (5.13.0-1008.10) impish; urgency=medium

  * impish/linux-oracle: 5.13.0-1008.10 -proposed tracker (LP: #1946331)

  * Miscellaneous Ubuntu changes
    - [Config] update toolchain version in configs after rebase

  [ Ubuntu: 5.13.0-19.19 ]

  * impish/linux: 5.13.0-19.19 -proposed tracker (LP: #1946337)
  * impish:linux-aws 5.13 panic during systemd autotest (LP: #1946001)
    - [Config] disable KFENCE

  [ Ubuntu: 5.13.0-18.18 ]

  * impish/linux: 5.13.0-18.18 -proposed tracker (LP: #1945995)
  * [21.10 FEAT] KVM: Use interpretation of specification exceptions
    (LP: #1932157)
    - KVM: s390: Enable specification exception interpretation

 -- Andrea Righi <email address hidden> Fri, 08 Oct 2021 17:26:41 +0200

Changed in linux-oracle (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Eugene Romanenko (eros2) wrote :

How to revert to old compression method for initramfs? Generating image with zstd is extremely slow and takes long time on slow CPU.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers