performance regression in dracut-install 060

[Viraniac]

[ Impact ]

When compared to Ubuntu 23.10, creating intramfs files with update-initramfs takes 2 to 5 times more time on ARM devices.

IIUC, dracut-install usage was added to initramfs-tools to speed up the process. But now its way slower. Even running update-initramfs on jammy, which doesn't use dracut-install, is way faster then the time taken on Noble.

first bad commit - https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870

Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble) with initramfs-tools 0.142ubuntu25.1 takes over six minutes:

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
  Range (min … max): 394.380 s … 411.445 s 10 runs
```

[ Test Plan ]

1. Measure `update-initramfs -u` before the update.
2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
3. update dracut-install / initramfs-tools-core
4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.

[ Where problems could occur ]

The code that is responsible for including the kernel modules into the initrd is touched. Negative consequences could be that some needed kernel modules will not be included any more (should be covered by the test case) or that building new initrds will fail.

The initramfs-tools fix changes how manual_add_modules behaves. `manual_add_modules` does not copy kernel modules, but queues them for being copied when the newly added function `apply_add_modules` is called.

I checked all instances of calls to `manual_add_modules` for possible regressions (see comment #15). Only miniramfs needs to be adjusted to also call `apply_add_modules`. But this change could break consumers of the `manual_add_modules` function that are outside of the Ubuntu archive. I googled for `apply_add_modules` but found no public outside users.

[ Other Info ]

$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04

$ apt-cache policy dracut-install
dracut-install:
  Installed: 060+5-1ubuntu3
  Candidate: 060+5-1ubuntu3
  Version table:
 *** 060+5-1ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages
        100 /var/lib/dpkg/status

[Benjamin Drung (bdrung)]

dracut-install is used in initramfs-tools to speed up the build time.

I tested `time update-initramfs -u` in chroots on my amd64 laptop. Results there:

* jammy: 15.585s
* mantic: 5.925s
* noble: 6.466s

So noble is a bit slower than mantic on my hardware. Is this slowdown hardware related or are all ARM devices affected? Can you provide some benchmark results and tested hardware results?

[Viraniac (viraniac)]

I have tested ubuntu server images on Khadas VIM4 and Raspberry Pi 4B 2GB variant.

VIM4
Jammy - 22s
Mantic - 21s
Noble - 1m 30s

RPi 4B
Mantic - 1m 50s
Noble - 3m 55s

I don't have a amd64 system to test ubuntu on at the moment.

[numbqq (numbqq)]

same issues on my arm devices.

[Dave Jones (waveform)]

Some results from some local testing. These tests were all performed on the same SD card (a Samsung EVO Select 64GB) with fresh installs of the jammy and noble server images, after running full upgrades and rebooting:

Pi 5, noble -- 01:23
Pi 4B, noble -- 03:05
Pi 3B+, noble -- 05:19

Pi 5, jammy -- unsupported
Pi 4B, jammy -- 01:06
Pi 3B+, jammy -- 01:59

So, there *appears* to be a substantial slow down from jammy. However, I don't think this is due to dracut or a performance regression in initramfs-tools. I remembered that, this cycle, the kernel team got rid of the linux-modules-extra split in the linux-raspi kernel. This results in a substantially increased initrd size because many more modules are included by default.

I took a listing of the jammy and noble initramfs contents and (after some manipulation to remove .zst suffixes, which are also new in the noble initramfs, and removing kernel version numbers). I'll attach the result of diff'ing these outputs, but a quick grep through the results and some calculations shows roughly 1000 extra files in the noble version.

In other words, this is not comparing like for like.

[Viraniac (viraniac)]

Dave Jones,

Could you please install dracut-install package from mantic on noble and rerun your tests?

Link to dracut-install package from mantic - http://ports.ubuntu.com/pool/main/d/dracut/dracut-install_059-4ubuntu2_arm64.deb

I have noticed that installing the above package significantly reduces time taken by update-initramfs and hence I think its a regression. Is there a way to configure the dracut-install that can reduce the time taken by the same?

[Dave Jones (waveform)]

@viraniac well, you're absolutely right! Same SD card with a fresh noble install, running on a Pi 4, first stock and second with a downgraded dracut from mantic:

Pi 4B, stock -- 03:05
Pi 4B, downgrade -- 01:10

That is a pretty substantial regression, and diff'ing the initrds once more showed they're basically identical in content (unsurprisingly), so it's not writing any less. I wonder if it's added some sync points or something similar.

[Benjamin Drung (bdrung)]

I marked thin-provisioning-tools, lvm2, and cryptsetup as affected to reduce the number of manual_add_modules calls in the initramfs-tools hooks in those packages. This will help a bit, but will probably not be enough to make it fast again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thin-provisioning-tools - 0.9.0-2ubuntu6

---------------
thin-provisioning-tools (0.9.0-2ubuntu6) oracular; urgency=medium

  * initramfs-hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:08:36 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.03.16-3ubuntu4

---------------
lvm2 (2.03.16-3ubuntu4) oracular; urgency=medium

  * initramfs-tools hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:42:08 +0200

[Benjamin Drung (bdrung)]

Stock noble on a Raspberry Pi Zero 2W:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
  Range (min … max): 409.139 s … 422.632 s 5 runs
```

Replace duplicate calls in thin-provisioning-tools, lvm2, and cryptsetup:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 375.805 s ± 5.753 s [User: 140.586 s, System: 218.345 s]
  Range (min … max): 369.914 s … 382.866 s 5 runs
```

Suggested further reduction of dracut-install calls via https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel:

```
$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 241.626 s ± 5.278 s [User: 60.018 s, System: 166.183 s]
  Range (min … max): 235.136 s … 249.194 s 5 runs
```

[Dave Jones (waveform)]

Results on a Pi 4B booting from SD card. Stock noble:

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 189.984 s ± 1.618 s [User: 75.720 s, System: 115.323 s]
  Range (min … max): 187.319 s … 191.142 s 5 runs

Then running the branch from https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel :

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 98.473 s ± 2.263 s [User: 26.061 s, System: 73.138 s]
  Range (min … max): 95.923 s … 101.560 s 5 runs

So that's a pretty substantial improvement. Still not *quite* at the mantic level, but it's in the same ball-park now, and that's not including the changes to lvm2 or cryptsetup.

[Benjamin Drung (bdrung)]

Submitted to Debian: https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/114

[Launchpad Janitor (janitor)]

This bug was fixed in the package cryptsetup - 2:2.7.0-1ubuntu5

---------------
cryptsetup (2:2.7.0-1ubuntu5) oracular; urgency=medium

  * initramfs hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:48:09 +0200

[Benjamin Drung (bdrung)]

https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/114 got merged after some iterations. This can be a breaking change. To restore the previous behavior, call `apply_add_modules` without arguments after a `manual_add_modules` call.

I checked all Ubuntu source packages that call manual_add_modules for possible regressions:

```
ac100-tarball-installer
amd64-microcode
aoetools
asahi-scripts
autopkgtest
bcachefs-tools
bcache-tools
bilibop
bootcd
brltty
casper
clevis
cloud-initramfs-tools
cryptsetup
dmraid
flashcache
fsprotect
fuse
fuse3
initramfs-tools
initramfs-tools-ubuntu-core
intel-microcode
librem-ec-acpi
live-boot
ltsp
lvm2
miniramfs
multipath-tools
mythbuntu-diskless
nbd
nvidia-graphics-drivers-384
olpc-xo1
open-infrastructure-system-boot
open-infrastructure-system-tools
open-iscsi
open-vm-tools
osk-sdl
r8168
rapiddisk
s390-tools
sysconfig
tcos
thin-provisioning-tools
unl0kr
v86d
zfcpdump-kernel
zfs-linux
```

amd64-microcode and initramfs-tools have following snippet:

```
if dpkg --compare-versions "${version}" lt 4.4 ; then
    manual_add_modules microcode && {
        # force_load has broken semanthics when the .ko file is missing
        find "${DESTDIR}/${MODULESDIR}" -type f -print | grep -qc '/microcode\.ko$' && {
          verbose "modular microcode driver detected"
          force_load microcode
        }
    }
fi
```

Ubuntu 16.04 LTS "Xenial Xerus" comes with Linux 4.4. Let's assume that we do not support kernel version < 4.4 in Ubuntu 24.04 LTS "Noble Numbat".

miniramfs just uses parts from initramfs-tools and need to call `apply_add_modules`.

[Benjamin Drung (bdrung)]

While checking those packages, I found hooks that do not support zstd-compressed kernel modules: bug #2068026

[Benjamin Drung (bdrung)]

I uploaded dracut, initramfs-tools, and miniramfs to oracular and the SRUs for noble. For easier testing, I also uploaded the noble SRU packages to https://launchpad.net/~bdrung/+archive/ubuntu/ppa

With the dracut and initramfs-tools SRUs the execution time on the Raspberry Pi Zero 2W reduces to:

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 207.655 s ± 7.033 s [User: 39.190 s, System: 156.799 s]
  Range (min … max): 191.754 s … 216.077 s 10 runs
```

[Launchpad Janitor (janitor)]

This bug was fixed in the package miniramfs - 1.0.2ubuntu1

---------------
miniramfs (1.0.2ubuntu1) oracular; urgency=medium

  * Call apply_add_modules after manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:58:24 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu28

---------------
initramfs-tools (0.142ubuntu28) oracular; urgency=medium

  * hook-functions: Use firmware search order from kernel
  * mkinitramfs: Resolve hidden dependencies after all modules were copied
  * reduce number of dracut-install calls. This can be a breaking change.
    To restore the previous behavior, call apply_add_modules without arguments
    after a manual_add_modules call. (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:26:39 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package dracut - 060+5-8ubuntu2

---------------
dracut (060+5-8ubuntu2) oracular; urgency=medium

  * perf(dracut-install): preload kmod resources for quicker module lookup
    (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:33:13 +0200

[Dave Jones (waveform)]

As requested, results from running stock noble on the same Pi 4B with the same SD card as before. First, stock noble (with all available upgrades):

$ sudo hyperfine --warmup 1 -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 193.558 s ± 2.334 s [User: 77.577 s, System: 118.253 s]
  Range (min … max): 190.964 s … 196.165 s 5 runs

Second, noble after update from the specified PPA (ppa:bdrung/ppa):

$ sudo hyperfine --warmup 1 -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 81.116 s ± 2.468 s [User: 16.125 s, System: 67.027 s]
  Range (min … max): 78.409 s … 84.017 s 5 runs

So a pretty substantial improvement :)

[Timo Aaltonen (tjaalton)]

Hello Viraniac, or anyone else affected,

Accepted dracut into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dracut/060+5-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted dracut (060+5-1ubuntu3.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

clevis/20-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#dracut

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Viraniac (viraniac)]

Tried on VIM4. It took 30s to build the initrd which is definitely way
better than 1m30s it was taking before. It still doesn't beat the 20s time
that is achievable just by reverting 131822e and 3de4c73 commits.

On Fri, Jun 14, 2024 at 8:01 PM Timo Aaltonen <email address hidden>
wrote:

> Hello Viraniac, or anyone else affected,
>
> Accepted dracut into noble-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/dracut/060+5-1ubuntu3.1 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-
> noble to verification-done-noble. If it does not fix the bug for you,
> please add a comment stating that, and change the tag to verification-
> failed-noble. In either case, without details of your testing we will
> not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: dracut (Ubuntu Noble)
> Status: New => Fix Committed
>
> ** Tags added: verification-needed verification-needed-noble
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2065180
>
> Title:
> performance regression in dracut-install 060
>
> Status in Dracut:
> New
> Status in cryptsetup package in Ubuntu:
> Fix Released
> Status in dracut package in Ubuntu:
> Fix Released
> Status in initramfs-tools package in Ubuntu:
> Fix Released
> Status in lvm2 package in Ubuntu:
> Fix Released
> Status in miniramfs package in Ubuntu:
> Fix Released
> Status in thin-provisioning-tools package in Ubuntu:
> Fix Released
> Status in cryptsetup source package in Noble:
> New
> Status in dracut source package in Noble:
> Fix Committed
> Status in initramfs-tools source package in Noble:
> New
> Status in lvm2 source package in Noble:
> New
> Status in miniramfs source package in Noble:
> New
> Status in thin-provisioning-tools source package in Noble:
> New
>
> Bug description:
> [ Impact ]
>
> When compared to Ubuntu 23.10, creating intramfs files with update-
> initramfs takes 2 to 5 times more time on ARM devices.
>
> IIUC, dracut-install usage was added to initramfs-tools to speed up
> the process. But now its way slower. Even running update-initramfs on
> jammy, which doesn't use dracut-install, is way faster then the time
> taken on Noble.
>
> first bad commit -
>
> https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f7803...