performance regression in dracut-install 060
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Dracut |
New
|
Unknown
|
|||
cloud-initramfs-tools (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned | ||
cryptsetup (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned | ||
dracut (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned | ||
initramfs-tools (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Benjamin Drung | ||
lvm2 (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned | ||
miniramfs (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
Noble |
New
|
Undecided
|
Unassigned | ||
open-iscsi (Ubuntu) |
Invalid
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned | ||
thin-provisioning-tools (Ubuntu) |
Fix Released
|
Undecided
|
Unassigned | ||
Noble |
Fix Released
|
Undecided
|
Unassigned |
Bug Description
[ Impact ]
When compared to Ubuntu 23.10, creating intramfs files with update-initramfs takes 2 to 5 times more time on ARM devices.
IIUC, dracut-install usage was added to initramfs-tools to speed up the process. But now its way slower. Even running update-initramfs on jammy, which doesn't use dracut-install, is way faster then the time taken on Noble.
first bad commit - https:/
Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble) with initramfs-tools 0.142ubuntu25.1 takes over six minutes:
```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
Range (min … max): 394.380 s … 411.445 s 10 runs
```
[ Test Plan ]
1. Measure `update-initramfs -u` before the update.
2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
3. update dracut-install / initramfs-
4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.
[ Where problems could occur ]
The code that is responsible for including the kernel modules into the initrd is touched. Negative consequences could be that some needed kernel modules will not be included any more (should be covered by the test case) or that building new initrds will fail.
The initramfs-tools fix changes how manual_add_modules behaves. `manual_
I checked all instances of calls to `manual_
[ Benchmarks ]
Stock noble on a Raspberry Pi Zero 2W:
```
bdrung@zero2w:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
Range (min … max): 409.139 s … 422.632 s 5 runs
```
noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on 2024-07-01):
```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 248.054 s ± 5.569 s [User: 67.410 s, System: 169.412 s]
Range (min … max): 238.909 s … 257.384 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1007 on 2024-07-11):
```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
Time (mean ± σ): 232.001 s ± 5.678 s [User: 55.456 s, System: 166.510 s]
Range (min … max): 222.120 s … 239.610 s 10 runs
```
noble with dracut-install 060+5-1ubuntu3.2 (with linux 6.8.0-1011.12 and all updates on 2024-09-19):
```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Time (mean ± σ): 219.439 s ± 4.869 s [User: 45.626 s, System: 163.172 s]
Range (min … max): 213.340 s … 227.411 s 10 runs
```
[ Reduce manual_add_modules calls ]
Besides making the dracut-install calls faster, group the dracut-install calls. Since the fix in oracular can cause regressions in custom hooks that rely on the current behavior, the SRU takes a safe approach which includes following packages (stating how many dracut-install calls are used):
* cryptsetup: 2 -> 1
* lvm2: 8 -> 1
* thin-provisioni
* open-iscsi: 9 -> 1
* cloud-initramfs
dracut-install calls on a Raspberry Pi Zero 2W:
| area | before | noble SRU | oracular |
|------
| auto_add_modules + apply_add_modules | 8 | 5 | 5 |
| calls by hooks + apply_add_modules | 42 | 20 | 2 |
| hidden_
| total | 51 | 26 | 8 |
[ Other Info ]
$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04
$ apt-cache policy dracut-install
dracut-install:
Installed: 060+5-1ubuntu3
Candidate: 060+5-1ubuntu3
Version table:
*** 060+5-1ubuntu3 500
500 http://
100 /var/lib/
Related branches
- Paride Legovini: Approve
-
Diff: 19 lines (+2/-6)1 file modifiedoverlayroot/hooks/overlayroot (+2/-6)
description: | updated |
summary: |
- performance regression in dracut-install + performance regression in dracut-install 060 |
Changed in dracut: | |
status: | Unknown → New |
description: | updated |
Changed in dracut (Ubuntu): | |
importance: | Undecided → High |
status: | New → Fix Committed |
Changed in miniramfs (Ubuntu): | |
importance: | Undecided → High |
status: | New → Fix Committed |
Changed in initramfs-tools (Ubuntu): | |
importance: | Undecided → High |
status: | New → Fix Committed |
tags: | added: fountations-todo |
Changed in initramfs-tools (Ubuntu Noble): | |
assignee: | nobody → Benjamin Drung (bdrung) |
tags: |
added: foundations-todo removed: fountations-todo |
description: | updated |
description: | updated |
Changed in open-iscsi (Ubuntu): | |
status: | New → Invalid |
Changed in cloud-initramfs-tools (Ubuntu): | |
status: | New → Fix Committed |
description: | updated |
Changed in initramfs-tools (Ubuntu Noble): | |
status: | Incomplete → New |
tags: | removed: foundations-todo |
tags: |
added: verification-done-noble removed: verification-needed-noble |
dracut-install is used in initramfs-tools to speed up the build time.
I tested `time update-initramfs -u` in chroots on my amd64 laptop. Results there:
* jammy: 15.585s
* mantic: 5.925s
* noble: 6.466s
So noble is a bit slower than mantic on my hardware. Is this slowdown hardware related or are all ARM devices affected? Can you provide some benchmark results and tested hardware results?