performance regression in dracut-install 060

[Viraniac]

When compared to Ubuntu 23.10, creating intramfs files with update-initramfs takes 2 to 5 times more time on ARM devices.

IIUC, dracut-install usage was added to initramfs-tools to speed up the process. But now its way slower. Even running update-initramfs on jammy, which doesn't use dracut-install, is way faster then the time taken on Noble.

first bad commit - https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870

$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04

$ apt-cache policy dracut-install
dracut-install:
  Installed: 060+5-1ubuntu3
  Candidate: 060+5-1ubuntu3
  Version table:
 *** 060+5-1ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages
        100 /var/lib/dpkg/status

[Benjamin Drung (bdrung)]

dracut-install is used in initramfs-tools to speed up the build time.

I tested `time update-initramfs -u` in chroots on my amd64 laptop. Results there:

* jammy: 15.585s
* mantic: 5.925s
* noble: 6.466s

So noble is a bit slower than mantic on my hardware. Is this slowdown hardware related or are all ARM devices affected? Can you provide some benchmark results and tested hardware results?

[Viraniac (viraniac)]

I have tested ubuntu server images on Khadas VIM4 and Raspberry Pi 4B 2GB variant.

VIM4
Jammy - 22s
Mantic - 21s
Noble - 1m 30s

RPi 4B
Mantic - 1m 50s
Noble - 3m 55s

I don't have a amd64 system to test ubuntu on at the moment.

[numbqq (numbqq)]

same issues on my arm devices.

[Dave Jones (waveform)]

Some results from some local testing. These tests were all performed on the same SD card (a Samsung EVO Select 64GB) with fresh installs of the jammy and noble server images, after running full upgrades and rebooting:

Pi 5, noble -- 01:23
Pi 4B, noble -- 03:05
Pi 3B+, noble -- 05:19

Pi 5, jammy -- unsupported
Pi 4B, jammy -- 01:06
Pi 3B+, jammy -- 01:59

So, there *appears* to be a substantial slow down from jammy. However, I don't think this is due to dracut or a performance regression in initramfs-tools. I remembered that, this cycle, the kernel team got rid of the linux-modules-extra split in the linux-raspi kernel. This results in a substantially increased initrd size because many more modules are included by default.

I took a listing of the jammy and noble initramfs contents and (after some manipulation to remove .zst suffixes, which are also new in the noble initramfs, and removing kernel version numbers). I'll attach the result of diff'ing these outputs, but a quick grep through the results and some calculations shows roughly 1000 extra files in the noble version.

In other words, this is not comparing like for like.

[Viraniac (viraniac)]

Dave Jones,

Could you please install dracut-install package from mantic on noble and rerun your tests?

Link to dracut-install package from mantic - http://ports.ubuntu.com/pool/main/d/dracut/dracut-install_059-4ubuntu2_arm64.deb

I have noticed that installing the above package significantly reduces time taken by update-initramfs and hence I think its a regression. Is there a way to configure the dracut-install that can reduce the time taken by the same?

[Dave Jones (waveform)]

@viraniac well, you're absolutely right! Same SD card with a fresh noble install, running on a Pi 4, first stock and second with a downgraded dracut from mantic:

Pi 4B, stock -- 03:05
Pi 4B, downgrade -- 01:10

That is a pretty substantial regression, and diff'ing the initrds once more showed they're basically identical in content (unsurprisingly), so it's not writing any less. I wonder if it's added some sync points or something similar.

[Benjamin Drung (bdrung)]

I marked thin-provisioning-tools, lvm2, and cryptsetup as affected to reduce the number of manual_add_modules calls in the initramfs-tools hooks in those packages. This will help a bit, but will probably not be enough to make it fast again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thin-provisioning-tools - 0.9.0-2ubuntu6

---------------
thin-provisioning-tools (0.9.0-2ubuntu6) oracular; urgency=medium

  * initramfs-hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:08:36 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.03.16-3ubuntu4

---------------
lvm2 (2.03.16-3ubuntu4) oracular; urgency=medium

  * initramfs-tools hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:42:08 +0200

[Benjamin Drung (bdrung)]

Stock noble on a Raspberry Pi Zero 2W:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
  Range (min … max): 409.139 s … 422.632 s 5 runs
```

Replace duplicate calls in thin-provisioning-tools, lvm2, and cryptsetup:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 375.805 s ± 5.753 s [User: 140.586 s, System: 218.345 s]
  Range (min … max): 369.914 s … 382.866 s 5 runs
```

Suggested further reduction of dracut-install calls via https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel:

```
$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 241.626 s ± 5.278 s [User: 60.018 s, System: 166.183 s]
  Range (min … max): 235.136 s … 249.194 s 5 runs
```

[Dave Jones (waveform)]

Results on a Pi 4B booting from SD card. Stock noble:

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 189.984 s ± 1.618 s [User: 75.720 s, System: 115.323 s]
  Range (min … max): 187.319 s … 191.142 s 5 runs

Then running the branch from https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel :

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 98.473 s ± 2.263 s [User: 26.061 s, System: 73.138 s]
  Range (min … max): 95.923 s … 101.560 s 5 runs

So that's a pretty substantial improvement. Still not *quite* at the mantic level, but it's in the same ball-park now, and that's not including the changes to lvm2 or cryptsetup.

[Benjamin Drung (bdrung)]

Submitted to Debian: https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/114