performance regression in dracut-install 060

[Viraniac]

[ Impact ]

When compared to Ubuntu 23.10, creating intramfs files with update-initramfs takes 2 to 5 times more time on ARM devices.

IIUC, dracut-install usage was added to initramfs-tools to speed up the process. But now its way slower. Even running update-initramfs on jammy, which doesn't use dracut-install, is way faster then the time taken on Noble.

first bad commit - https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870

Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble) with initramfs-tools 0.142ubuntu25.1 takes over six minutes:

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System: 228.909 s]
  Range (min … max): 394.380 s … 411.445 s 10 runs
```

[ Test Plan ]

1. Measure `update-initramfs -u` before the update.
2. Log the content of the initrd before the update: `lsinitramfs /boot/initrd.img`
3. update dracut-install / initramfs-tools-core
4. Measure `update-initramfs -u`. It should be faster (the performance improvements on amd64 should be very small and might be within the measurement uncertainty).
5. Check with lsinitramfs that the content of the newly generated initrd hasn't changed.

[ Where problems could occur ]

The code that is responsible for including the kernel modules into the initrd is touched. Negative consequences could be that some needed kernel modules will not be included any more (should be covered by the test case) or that building new initrds will fail.

The initramfs-tools fix changes how manual_add_modules behaves. `manual_add_modules` does not copy kernel modules, but queues them for being copied when the newly added function `apply_add_modules` is called.

I checked all instances of calls to `manual_add_modules` for possible regressions (see comment #15). Only miniramfs needs to be adjusted to also call `apply_add_modules`. But this change could break consumers of the `manual_add_modules` function that are outside of the Ubuntu archive. I googled for `apply_add_modules` but found no public outside users.

[ Benchmarks ]

Stock noble on a Raspberry Pi Zero 2W:

```
bdrung@zero2w:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
  Range (min … max): 409.139 s … 422.632 s 5 runs
```

noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on 2024-07-01):

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 248.054 s ± 5.569 s [User: 67.410 s, System: 169.412 s]
  Range (min … max): 238.909 s … 257.384 s 10 runs
```

[ Reduce manual_add_modules calls ]

Besides making the dracut-install calls faster, group the dracut-install calls. Since the fix in oracular can cause regressions in custom hooks that rely on the current behavior, the SRU takes a safe approach which includes following packages (stating how many dracut-install calls are used):

 * cryptsetup: 2 -> 1
 * lvm2: 8 -> 1
 * thin-provisioning-tools: 3 -> 1
 * open-iscsi: 9 -> 1
 * cloud-initramfs-tools: 5 -> 1

dracut-install calls on a Raspberry Pi Zero 2W:

| area | before | noble SRU | oracular |
|--------------------------------------|--------|-----------|----------|
| auto_add_modules + apply_add_modules | 8 | 5 | 5 |
| calls by hooks + apply_add_modules | 42 | 20 | 2 |
| hidden_dep_add_modules | 1 | 1 | 1 |
| total | 51 | 26 | 8 |

[ Other Info ]

$ lsb_release -rd
No LSB modules are available.
Description: Ubuntu 24.04 LTS
Release: 24.04

$ apt-cache policy dracut-install
dracut-install:
  Installed: 060+5-1ubuntu3
  Candidate: 060+5-1ubuntu3
  Version table:
 *** 060+5-1ubuntu3 500
        500 http://ports.ubuntu.com/ubuntu-ports noble/main arm64 Packages
        100 /var/lib/dpkg/status

[Benjamin Drung (bdrung)]

dracut-install is used in initramfs-tools to speed up the build time.

I tested `time update-initramfs -u` in chroots on my amd64 laptop. Results there:

* jammy: 15.585s
* mantic: 5.925s
* noble: 6.466s

So noble is a bit slower than mantic on my hardware. Is this slowdown hardware related or are all ARM devices affected? Can you provide some benchmark results and tested hardware results?

[Viraniac (viraniac)]

I have tested ubuntu server images on Khadas VIM4 and Raspberry Pi 4B 2GB variant.

VIM4
Jammy - 22s
Mantic - 21s
Noble - 1m 30s

RPi 4B
Mantic - 1m 50s
Noble - 3m 55s

I don't have a amd64 system to test ubuntu on at the moment.

[numbqq (numbqq)]

same issues on my arm devices.

[Dave Jones (waveform)]

Some results from some local testing. These tests were all performed on the same SD card (a Samsung EVO Select 64GB) with fresh installs of the jammy and noble server images, after running full upgrades and rebooting:

Pi 5, noble -- 01:23
Pi 4B, noble -- 03:05
Pi 3B+, noble -- 05:19

Pi 5, jammy -- unsupported
Pi 4B, jammy -- 01:06
Pi 3B+, jammy -- 01:59

So, there *appears* to be a substantial slow down from jammy. However, I don't think this is due to dracut or a performance regression in initramfs-tools. I remembered that, this cycle, the kernel team got rid of the linux-modules-extra split in the linux-raspi kernel. This results in a substantially increased initrd size because many more modules are included by default.

I took a listing of the jammy and noble initramfs contents and (after some manipulation to remove .zst suffixes, which are also new in the noble initramfs, and removing kernel version numbers). I'll attach the result of diff'ing these outputs, but a quick grep through the results and some calculations shows roughly 1000 extra files in the noble version.

In other words, this is not comparing like for like.

[Viraniac (viraniac)]

Dave Jones,

Could you please install dracut-install package from mantic on noble and rerun your tests?

Link to dracut-install package from mantic - http://ports.ubuntu.com/pool/main/d/dracut/dracut-install_059-4ubuntu2_arm64.deb

I have noticed that installing the above package significantly reduces time taken by update-initramfs and hence I think its a regression. Is there a way to configure the dracut-install that can reduce the time taken by the same?

[Dave Jones (waveform)]

@viraniac well, you're absolutely right! Same SD card with a fresh noble install, running on a Pi 4, first stock and second with a downgraded dracut from mantic:

Pi 4B, stock -- 03:05
Pi 4B, downgrade -- 01:10

That is a pretty substantial regression, and diff'ing the initrds once more showed they're basically identical in content (unsurprisingly), so it's not writing any less. I wonder if it's added some sync points or something similar.

[Benjamin Drung (bdrung)]

I marked thin-provisioning-tools, lvm2, and cryptsetup as affected to reduce the number of manual_add_modules calls in the initramfs-tools hooks in those packages. This will help a bit, but will probably not be enough to make it fast again.

[Launchpad Janitor (janitor)]

This bug was fixed in the package thin-provisioning-tools - 0.9.0-2ubuntu6

---------------
thin-provisioning-tools (0.9.0-2ubuntu6) oracular; urgency=medium

  * initramfs-hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:08:36 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package lvm2 - 2.03.16-3ubuntu4

---------------
lvm2 (2.03.16-3ubuntu4) oracular; urgency=medium

  * initramfs-tools hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:42:08 +0200

[Benjamin Drung (bdrung)]

Stock noble on a Raspberry Pi Zero 2W:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 415.664 s ± 6.015 s [User: 166.728 s, System: 232.523 s]
  Range (min … max): 409.139 s … 422.632 s 5 runs
```

Replace duplicate calls in thin-provisioning-tools, lvm2, and cryptsetup:

```
bdrung@zero3:~$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 375.805 s ± 5.753 s [User: 140.586 s, System: 218.345 s]
  Range (min … max): 369.914 s … 382.866 s 5 runs
```

Suggested further reduction of dracut-install calls via https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel:

```
$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 241.626 s ± 5.278 s [User: 60.018 s, System: 166.183 s]
  Range (min … max): 235.136 s … 249.194 s 5 runs
```

[Dave Jones (waveform)]

Results on a Pi 4B booting from SD card. Stock noble:

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 189.984 s ± 1.618 s [User: 75.720 s, System: 115.323 s]
  Range (min … max): 187.319 s … 191.142 s 5 runs

Then running the branch from https://code.launchpad.net/~bdrung/ubuntu/+source/initramfs-tools/+git/initramfs-tools/+ref/ubuntu/devel :

$ sudo hyperfine -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 98.473 s ± 2.263 s [User: 26.061 s, System: 73.138 s]
  Range (min … max): 95.923 s … 101.560 s 5 runs

So that's a pretty substantial improvement. Still not *quite* at the mantic level, but it's in the same ball-park now, and that's not including the changes to lvm2 or cryptsetup.

[Benjamin Drung (bdrung)]

Submitted to Debian: https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/114

[Launchpad Janitor (janitor)]

This bug was fixed in the package cryptsetup - 2:2.7.0-1ubuntu5

---------------
cryptsetup (2:2.7.0-1ubuntu5) oracular; urgency=medium

  * initramfs hook: Combine calls to manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Fri, 24 May 2024 09:48:09 +0200

[Benjamin Drung (bdrung)]

https://salsa.debian.org/kernel-team/initramfs-tools/-/merge_requests/114 got merged after some iterations. This can be a breaking change. To restore the previous behavior, call `apply_add_modules` without arguments after a `manual_add_modules` call.

I checked all Ubuntu source packages that call manual_add_modules for possible regressions:

```
ac100-tarball-installer
amd64-microcode
aoetools
asahi-scripts
autopkgtest
bcachefs-tools
bcache-tools
bilibop
bootcd
brltty
casper
clevis
cloud-initramfs-tools
cryptsetup
dmraid
flashcache
fsprotect
fuse
fuse3
initramfs-tools
initramfs-tools-ubuntu-core
intel-microcode
librem-ec-acpi
live-boot
ltsp
lvm2
miniramfs
multipath-tools
mythbuntu-diskless
nbd
nvidia-graphics-drivers-384
olpc-xo1
open-infrastructure-system-boot
open-infrastructure-system-tools
open-iscsi
open-vm-tools
osk-sdl
r8168
rapiddisk
s390-tools
sysconfig
tcos
thin-provisioning-tools
unl0kr
v86d
zfcpdump-kernel
zfs-linux
```

amd64-microcode and initramfs-tools have following snippet:

```
if dpkg --compare-versions "${version}" lt 4.4 ; then
    manual_add_modules microcode && {
        # force_load has broken semanthics when the .ko file is missing
        find "${DESTDIR}/${MODULESDIR}" -type f -print | grep -qc '/microcode\.ko$' && {
          verbose "modular microcode driver detected"
          force_load microcode
        }
    }
fi
```

Ubuntu 16.04 LTS "Xenial Xerus" comes with Linux 4.4. Let's assume that we do not support kernel version < 4.4 in Ubuntu 24.04 LTS "Noble Numbat".

miniramfs just uses parts from initramfs-tools and need to call `apply_add_modules`.

[Benjamin Drung (bdrung)]

While checking those packages, I found hooks that do not support zstd-compressed kernel modules: bug #2068026

[Benjamin Drung (bdrung)]

I uploaded dracut, initramfs-tools, and miniramfs to oracular and the SRUs for noble. For easier testing, I also uploaded the noble SRU packages to https://launchpad.net/~bdrung/+archive/ubuntu/ppa

With the dracut and initramfs-tools SRUs the execution time on the Raspberry Pi Zero 2W reduces to:

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 207.655 s ± 7.033 s [User: 39.190 s, System: 156.799 s]
  Range (min … max): 191.754 s … 216.077 s 10 runs
```

[Launchpad Janitor (janitor)]

This bug was fixed in the package miniramfs - 1.0.2ubuntu1

---------------
miniramfs (1.0.2ubuntu1) oracular; urgency=medium

  * Call apply_add_modules after manual_add_modules (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:58:24 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package initramfs-tools - 0.142ubuntu28

---------------
initramfs-tools (0.142ubuntu28) oracular; urgency=medium

  * hook-functions: Use firmware search order from kernel
  * mkinitramfs: Resolve hidden dependencies after all modules were copied
  * reduce number of dracut-install calls. This can be a breaking change.
    To restore the previous behavior, call apply_add_modules without arguments
    after a manual_add_modules call. (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:26:39 +0200

[Launchpad Janitor (janitor)]

This bug was fixed in the package dracut - 060+5-8ubuntu2

---------------
dracut (060+5-8ubuntu2) oracular; urgency=medium

  * perf(dracut-install): preload kmod resources for quicker module lookup
    (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 16:33:13 +0200

[Dave Jones (waveform)]

As requested, results from running stock noble on the same Pi 4B with the same SD card as before. First, stock noble (with all available upgrades):

$ sudo hyperfine --warmup 1 -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 193.558 s ± 2.334 s [User: 77.577 s, System: 118.253 s]
  Range (min … max): 190.964 s … 196.165 s 5 runs

Second, noble after update from the specified PPA (ppa:bdrung/ppa):

$ sudo hyperfine --warmup 1 -r 5 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 81.116 s ± 2.468 s [User: 16.125 s, System: 67.027 s]
  Range (min … max): 78.409 s … 84.017 s 5 runs

So a pretty substantial improvement :)

[Timo Aaltonen (tjaalton)]

Hello Viraniac, or anyone else affected,

Accepted dracut into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dracut/060+5-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted dracut (060+5-1ubuntu3.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

clevis/20-1 (ppc64el)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#dracut

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Viraniac (viraniac)]

Tried on VIM4. It took 30s to build the initrd which is definitely way
better than 1m30s it was taking before. It still doesn't beat the 20s time
that is achievable just by reverting 131822e and 3de4c73 commits.

On Fri, Jun 14, 2024 at 8:01 PM Timo Aaltonen <email address hidden>
wrote:

> Hello Viraniac, or anyone else affected,
>
> Accepted dracut into noble-proposed. The package will build now and be
> available at
> https://launchpad.net/ubuntu/+source/dracut/060+5-1ubuntu3.1 in a few
> hours, and then in the -proposed repository.
>
> Please help us by testing this new package. See
> https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how
> to enable and use -proposed. Your feedback will aid us getting this
> update out to other Ubuntu users.
>
> If this package fixes the bug for you, please add a comment to this bug,
> mentioning the version of the package you tested, what testing has been
> performed on the package and change the tag from verification-needed-
> noble to verification-done-noble. If it does not fix the bug for you,
> please add a comment stating that, and change the tag to verification-
> failed-noble. In either case, without details of your testing we will
> not be able to proceed.
>
> Further information regarding the verification process can be found at
> https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in
> advance for helping!
>
> N.B. The updated package will be released to -updates after the bug(s)
> fixed by this package have been verified and the package has been in
> -proposed for a minimum of 7 days.
>
> ** Changed in: dracut (Ubuntu Noble)
> Status: New => Fix Committed
>
> ** Tags added: verification-needed verification-needed-noble
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2065180
>
> Title:
> performance regression in dracut-install 060
>
> Status in Dracut:
> New
> Status in cryptsetup package in Ubuntu:
> Fix Released
> Status in dracut package in Ubuntu:
> Fix Released
> Status in initramfs-tools package in Ubuntu:
> Fix Released
> Status in lvm2 package in Ubuntu:
> Fix Released
> Status in miniramfs package in Ubuntu:
> Fix Released
> Status in thin-provisioning-tools package in Ubuntu:
> Fix Released
> Status in cryptsetup source package in Noble:
> New
> Status in dracut source package in Noble:
> Fix Committed
> Status in initramfs-tools source package in Noble:
> New
> Status in lvm2 source package in Noble:
> New
> Status in miniramfs source package in Noble:
> New
> Status in thin-provisioning-tools source package in Noble:
> New
>
> Bug description:
> [ Impact ]
>
> When compared to Ubuntu 23.10, creating intramfs files with update-
> initramfs takes 2 to 5 times more time on ARM devices.
>
> IIUC, dracut-install usage was added to initramfs-tools to speed up
> the process. But now its way slower. Even running update-initramfs on
> jammy, which doesn't use dracut-install, is way faster then the time
> taken on Noble.
>
> first bad commit -
>
> https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f7803...

[Benjamin Drung (bdrung)]

Thanks for the test on the VIM4. Can you also verify that the content of the initrd hasn't change (see "Test Plan" in the bug description)?

Further speedup will be achieved by the initramfs-tools change (that is waiting in the SRU upload queue).

[Viraniac (viraniac)]

Sorry forgot to mention about the contents before. I did checked the
contents. They were exactly the same.

On Mon, Jun 17, 2024 at 6:01 PM Benjamin Drung <email address hidden>
wrote:

> Thanks for the test on the VIM4. Can you also verify that the content of
> the initrd hasn't change (see "Test Plan" in the bug description)?
>
> Further speedup will be achieved by the initramfs-tools change (that is
> waiting in the SRU upload queue).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2065180
>
> Title:
> performance regression in dracut-install 060
>
> Status in Dracut:
> New
> Status in cryptsetup package in Ubuntu:
> Fix Released
> Status in dracut package in Ubuntu:
> Fix Released
> Status in initramfs-tools package in Ubuntu:
> Fix Released
> Status in lvm2 package in Ubuntu:
> Fix Released
> Status in miniramfs package in Ubuntu:
> Fix Released
> Status in thin-provisioning-tools package in Ubuntu:
> Fix Released
> Status in cryptsetup source package in Noble:
> New
> Status in dracut source package in Noble:
> Fix Committed
> Status in initramfs-tools source package in Noble:
> New
> Status in lvm2 source package in Noble:
> New
> Status in miniramfs source package in Noble:
> New
> Status in thin-provisioning-tools source package in Noble:
> New
>
> Bug description:
> [ Impact ]
>
> When compared to Ubuntu 23.10, creating intramfs files with update-
> initramfs takes 2 to 5 times more time on ARM devices.
>
> IIUC, dracut-install usage was added to initramfs-tools to speed up
> the process. But now its way slower. Even running update-initramfs on
> jammy, which doesn't use dracut-install, is way faster then the time
> taken on Noble.
>
> first bad commit -
>
> https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870
>
> Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble)
> with initramfs-tools 0.142ubuntu25.1 takes over six minutes:
>
> ```
> bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
> Benchmark 1: update-initramfs -u
> Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System:
> 228.909 s]
> Range (min … max): 394.380 s … 411.445 s 10 runs
> ```
>
> [ Test Plan ]
>
> 1. Measure `update-initramfs -u` before the update.
> 2. Log the content of the initrd before the update: `lsinitramfs
> /boot/initrd.img`
> 3. update dracut-install / initramfs-tools-core
> 4. Measure `update-initramfs -u`. It should be faster (the performance
> improvements on amd64 should be very small and might be within the
> measurement uncertainty).
> 5. Check with lsinitramfs that the content of the newly generated initrd
> hasn't changed.
>
> [ Where problems could occur ]
>
> The code that is responsible for including the kernel modules into the
> initrd is touched. Negative consequences could be that some needed
> kernel modules will not be included any more (should be covered by the
> test case) or that building new initrds will fail.
>
> The initramfs-tools fix changes how manual_add_modules behaves...

[Benjamin Drung (bdrung)]

Thanks. So marking it as verification-done-noble for dracut.

[Chris Halse Rogers (raof)]

Correct me if I'm wrong, but isn't `manual_add_modules` exposed to user-configuration? User configuration in /etc/initramfs-tools/hooks is expected to include /usr/share/initramfs-tools/hook-functions, and so making this API break has the possibility of causing currently working user configurations to fail, resulting in unbootable systems?

I don't think you can change the behaviour of `manual_add_modules` this way in an SRU.

Alternatively, could you change it so that `apply_add_modules` is run exactly once, after all the hooks have been run?

[Benjamin Drung (bdrung)]

`manual_add_modules` is exposed to user-configuration. We run `apply_add_modules` after the hooks has been run (CONFDIR is the user configuration directory):

```
run_scripts_optional "${CONFDIR}"/hooks

# cache boot run order
for b in $(cd "${DESTDIR}/scripts" && find . -mindepth 1 -type d); do
 cache_run_scripts "${DESTDIR}" "/scripts/${b#./}"
done

apply_add_modules
```

The possible break appears when the user calls `manual_add_modules` and expects the modules to be present in $DESTDIR afterwards, but the modules will only be copied after apply_add_modules is called.

[Chris Halse Rogers (raof)]

Ooooh, right. The hook might want to call something like `depmod` against the modules.

So, that's a reasonable thing for a user-configured hook to do, which means we can't break it in an SRU. It *would* still be nice to get the significant speedup you've got here, though. Could this change to `manual_add_modules` *only* apply to hooks installed by packages (ie: in /usr/share/initramfs-tools)? Those we *can* fix.

[Benjamin Drung (bdrung)]

I don't know how to differentiate between if `manual_add_modules` was called by a script in /usr/share/initramfs-tools or from outside.

The only safer solution that I can come up with: Keep `manual_add_modules` as it is and introduce a new function (e.g. `manual_stage_modules`) that introduces the new behavior. The consumers in initramfs-tools will switch to the new function. Then the worst offenders (that call `manual_add_modules` many times) need a SRU to change from `manual_add_modules` to `manual_stage_modules`. initramfs-tools in oracular would get a `manual_stage_modules` function as well for easier upgrades.

Then there will be only a slight risk left: Custom scripts that rely on other hooks (that switched from `manual_add_modules` to `manual_stage_modules`) to have the kernel modules copies to $DESTDIR.

[Launchpad Janitor (janitor)]

This bug was fixed in the package dracut - 060+5-1ubuntu3.1

---------------
dracut (060+5-1ubuntu3.1) noble; urgency=medium

  * perf(dracut-install): preload kmod resources for quicker module lookup
    (LP: #2065180)

 -- Benjamin Drung <email address hidden> Tue, 04 Jun 2024 17:21:56 +0200

[Andreas Hasenack (ahasenack)]

The verification of the Stable Release Update for dracut has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Benjamin Drung (bdrung)]

There are 51 dracut-install calls on my Raspberry Pi Zero 2W on Ubuntu 24.04:

```
dracut-install -m -P /hid-(a4tech|cypress|dr|elecom|gyration|icade|kensington|kye|lcpower|magicmouse|ntrig|petalynx|picolcd|pl|ps3remote|quanta|roccat-ko.*|roccat-pyra|saitek|sensor-hub|sony|speedlink|tivo|twinhan|uclogic|wacom|waltop|wiimote|zydacron|.*ff)\.ko =drivers/hid
dracut-install -m =drivers/usb/host -P /(hwa-hc|sl811_cs|sl811-hcd|u132-hcd|whci-hcd)\.ko
dracut-install -m -P /((cdc_mbim|ipheth|qmi_wwan|sierra_net|veth|xen-netback)\.ko|(isdn|net/ethernet|net/phy|net/team|uwb|wan|wireless)/) -s eth_type_trans|register_virtio_device|usbnet_open =drivers/net
dracut-install -m -s ahci_platform_get_resources|ata_scsi_ioctl|scsi_add_host|blk_cleanup_queue|register_mtd_blktrans|scsi_esp_register|register_virtio_device|usb_stor_disconnect|mmc_add_host|sdhci_add_host|scsi_add_host_with_dma|blk_mq_alloc_disk|blk_mq_alloc_request|blk_mq_destroy_queue|blk_cleanup_disk|dw_mc_probe|dw_mci_pltfm_register|nvme_init_ctrl|iscsi_register_transport =drivers/scsi =drivers/ufs
dracut-install -m -s ahci_platform_get_resources|ata_scsi_ioctl|scsi_add_host|blk_cleanup_queue|register_mtd_blktrans|scsi_esp_register|register_virtio_device|usb_stor_disconnect|mmc_add_host|sdhci_add_host|scsi_add_host_with_dma|blk_mq_alloc_disk|blk_mq_alloc_request|blk_mq_destroy_queue|blk_cleanup_disk|dw_mc_probe|dw_mci_pltfm_register|nvme_init_ctrl =drivers/block =drivers/nvme =drivers/dax vmd
dracut-install -m -s nvdimm_bus_register =drivers/nvdimm =drivers/acpi
dracut-install -m -s ahci_platform_get_resources|ata_scsi_ioctl|scsi_add_host|blk_cleanup_queue|register_mtd_blktrans|scsi_esp_register|register_virtio_device|usb_stor_disconnect|mmc_add_host|sdhci_add_host|scsi_add_host_with_dma|blk_mq_alloc_disk|blk_mq_alloc_request|blk_mq_destroy_queue|blk_cleanup_disk|dw_mc_probe|dw_mci_pltfm_register|nvme_init_ctrl =drivers/ata
dracut-install -m btrfs ext2 ext3 ext4 f2fs isofs jfs reiserfs squashfs udf xfs nfs nfsv2 nfsv3 nfsv4 af_packet atkbd i8042 psmouse virtio_pci virtio_mmio vfat nls_cp437 nls_iso8859-1 ehci-hcd ehci-pci ehci-platform ohci-hcd ohci-pci uhci-hcd usbhid xhci-hcd xhci-pci xhci-plat-hcd =drivers/usb/typec =drivers/usb/c67x00 =drivers/usb/renesas_usbhs extcon-usb-gpio extcon-usbc-cros-ec =drivers/input/keyboard cros_ec_spi intel_lpss_pci spi_pxa2xx_platform surface_aggregator_registry =drivers/tty/serial =drivers/bus =drivers/i2c/muxes =drivers/pci/controller =drivers/pinctrl =drivers/clk =drivers/i2c/busses =drivers/gpio =drivers/mfd =drivers/nvmem =drivers/phy =drivers/power =drivers/regulator =drivers/reset =drivers/spi =drivers/spmi =drivers/soc =drivers/usb/chipidea =drivers/usb/dwc2 =drivers/usb/dwc3 =drivers/usb/isp1760 =drivers/usb/musb =drivers/usb/phy =drivers/rtc axp20x_usb_power =drivers/char/hw_random =drivers/net/ethernet =drivers/net/mdio =drivers/net/phy 8021q ipvlan =drivers/ide be2iscsi bnx2i cxgb3i cxgb4i qedi qla4xxx scsi_dh_alua scsi_dh_emc scsi_dh_rdac mptfc mptsas mptscsih mptspi zfcp scsi_transport_srp dax_pmem nd_pmem dasd_diag_mod dasd_eckd_mod dasd_fba_mod firewire-ohci firewire-sbp2 =drivers/mmc =drivers/usb/storage rockchipdrm pwm-cros-ec pwm_bl pwm-rockch...

[Benjamin Drung (bdrung)]

There is another performance improvement upstream: https://github.com/dracut-ng/dracut-ng/pull/408

I tested this change a Raspberry Pi Zero 2W, but it had no measurable performance improvement:

```
$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 249.595 s ± 7.243 s [User: 66.584 s, System: 170.342 s]
  Range (min … max): 240.879 s … 260.506 s 10 runs
```

Dave, can you test dracut 060+5-1ubuntu3.2~ppa1 from https://launchpad.net/~bdrung/+archive/ubuntu/ppa to see if that would improve the situation on the other Pis? Viraniac, could you test that version on your VIM4?

[Viraniac (viraniac)]

Hi Benjamin,

I am also not seeing much difference on VIM4. The dracut-install version
060+5-1ubuntu3.2~ppa1 is providing only 200-300 ms improvement over
060+5-1ubuntu3.1 version.

On Tue, Jul 2, 2024 at 12:10 AM Benjamin Drung <email address hidden>
wrote:

> There is another performance improvement upstream:
> https://github.com/dracut-ng/dracut-ng/pull/408
>
> I tested this change a Raspberry Pi Zero 2W, but it had no measurable
> performance improvement:
>
> ```
> $ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
> Benchmark 1: update-initramfs -u
> Time (mean ± σ): 249.595 s ± 7.243 s [User: 66.584 s, System:
> 170.342 s]
> Range (min … max): 240.879 s … 260.506 s 10 runs
> ```
>
> Dave, can you test dracut 060+5-1ubuntu3.2~ppa1 from
> https://launchpad.net/~bdrung/+archive/ubuntu/ppa to see if that would
> improve the situation on the other Pis? Viraniac, could you test that
> version on your VIM4?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/2065180
>
> Title:
> performance regression in dracut-install 060
>
> Status in Dracut:
> New
> Status in cryptsetup package in Ubuntu:
> Fix Released
> Status in dracut package in Ubuntu:
> Fix Released
> Status in initramfs-tools package in Ubuntu:
> Fix Released
> Status in lvm2 package in Ubuntu:
> Fix Released
> Status in miniramfs package in Ubuntu:
> Fix Released
> Status in thin-provisioning-tools package in Ubuntu:
> Fix Released
> Status in cryptsetup source package in Noble:
> New
> Status in dracut source package in Noble:
> Fix Released
> Status in initramfs-tools source package in Noble:
> Incomplete
> Status in lvm2 source package in Noble:
> New
> Status in miniramfs source package in Noble:
> New
> Status in thin-provisioning-tools source package in Noble:
> New
>
> Bug description:
> [ Impact ]
>
> When compared to Ubuntu 23.10, creating intramfs files with update-
> initramfs takes 2 to 5 times more time on ARM devices.
>
> IIUC, dracut-install usage was added to initramfs-tools to speed up
> the process. But now its way slower. Even running update-initramfs on
> jammy, which doesn't use dracut-install, is way faster then the time
> taken on Noble.
>
> first bad commit -
>
> https://github.com/dracutdevs/dracut/commit/3de4c7313260fb600507c9b87f780390b874c870
>
> Updating the initrd on a Raspberry Pi Zero 2W on Ubuntu 24.04 (noble)
> with initramfs-tools 0.142ubuntu25.1 takes over six minutes:
>
> ```
> bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
> Benchmark 1: update-initramfs -u
> Time (mean ± σ): 402.751 s ± 5.592 s [User: 166.316 s, System:
> 228.909 s]
> Range (min … max): 394.380 s … 411.445 s 10 runs
> ```
>
> [ Test Plan ]
>
> 1. Measure `update-initramfs -u` before the update.
> 2. Log the content of the initrd before the update: `lsinitramfs
> /boot/initrd.img`
> 3. update dracut-install / initramfs-tools-core
> 4. Measure `update-initramfs -u`. It should be faster (the performance
> improvements on amd64 should be very small and might be within the...

[Benjamin Drung (bdrung)]

The second iteration of the noble SRU: Reduce the number of dracut-install calls from 51 down to 26. This involves adjusting six packages: cryptsetup (2 -> 1), lvm2 (8 -> 1), thin-provisioning-tools (3 -> 1), open-iscsi (9 -> 1), cloud-initramfs-tools (5 -> 1), and initramfs-tools itself (8 -> 5).

Benchmark result on Raspberry Pi Zero 2W: Ubuntu noble with dracut-install 060+5-1ubuntu3.1 (with linux 6.8.0-1006.6 on 2024-07-02) with those changes mentioned above:

```
bdrung@zero2w:~$ sudo hyperfine --warmup 1 -r 10 "update-initramfs -u"
Benchmark 1: update-initramfs -u
  Time (mean ± σ): 223.113 s ± 5.167 s [User: 50.701 s, System: 159.711 s]
  Range (min … max): 215.693 s … 230.826 s 10 runs
```

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-initramfs-tools - 0.49

---------------
cloud-initramfs-tools (0.49) oracular; urgency=medium

  [ Benjamin Drung ]
  * overlayroot: Combine calls to manual_add_modules (LP: #2065180)

 -- Paride Legovini <email address hidden> Tue, 02 Jul 2024 12:38:14 +0200

[Chris Halse Rogers (raof)]

> I don't know how to differentiate between if `manual_add_modules` was called by a script in /usr/share/initramfs-tools or from outside.

I *think* you could have `manual_add_modules` check the environment for something like `INITRAMFS_TOOLS_IN_USER_CONFIG=1` or something, and change behaviour based on that. You'd not set it while executing files in /usr/share/initramfs-tools, then you'd set it before executing the scripts in user config directories.

> The only safer solution that I can come up with: Keep `manual_add_modules` as it is and introduce a new function (e.g. `manual_stage_modules`) that introduces the new behavior.

This would also work. This approach has the advantage that it's clearer what's happening, and the disadvantages that it needs more work in other packages to get the benefit and that it still changes the behaviour in ways which theoretically might be visible to user configuration.

I don't have a strong opinion on which approach should be taken (particularly: I've not *actually implemented* my solution, so it might be infeasible).

[Benjamin Drung (bdrung)]

The problem is that a custom hook might rely on the behavior that all kernel modules were copied to $DESTDIR.

For the SRU I am playing the safe card now as documented in the "Reduce manual_add_modules calls" section of the bug description.

[Benjamin Drung (bdrung)]

Another dracut-install commit that we can pick: https://github.com/dracut-ng/dracut-ng/pull/479

[Benjamin Drung (bdrung)]

Resetting dracut since there are more performance fixes for dracut-install

[Launchpad Janitor (janitor)]

This bug was fixed in the package dracut - 102-3ubuntu4

---------------
dracut (102-3ubuntu4) oracular; urgency=medium

  * Cherry-pick upstream fixes for systemd 256 (LP: #2069290):
    - fix(test): use --add instead of --modules to create test-makeroot
    - test: avoid writing to rootfs as it might be read-only
  * fix(dracut-initramfs-restore.sh): correct initrd globbing
  * feat(lsinitrd.sh): support configurable initrd filenames
  * Default initrdname to initrd.img-${kernel}
  * Cherry-pick upstream performance fixes (LP: #2065180):
    - perf(dracut-install): memoize find_kmod_module_from_sysfs_node
    - perf(dracut-install): use driver/module sysfs dirs for module name
  * test: virtual hardware watchdog not available on s390x

 -- Benjamin Drung <email address hidden> Mon, 08 Jul 2024 20:37:51 +0200

[Chris Halse Rogers (raof)]

I'm not a bash expert by any means, but something like this would appear to work?
```
function manual_add_modules() {
   ... normal stuff goes here ...
   if [ $IN_USER_CONFIG -gt 0 ]; then
      apply_add_modules
   fi
}
...

in mkinitramfs:

...
IN_USER_CONFIG=0

...

apply_add_modules
run_scripts_optional /usr/share/initramfs-tools/hooks
apply_add_modules
IN_USER_CONFIG=1
run_scripts_optional "${CONFDIR}"/hooks...
...
```

(I am also reviewing the existing upload)

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted initramfs-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/initramfs-tools/0.142ubuntu25.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted cryptsetup into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cryptsetup/2:2.7.0-1ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted lvm2 into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/lvm2/2.03.16-3ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted thin-provisioning-tools into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/thin-provisioning-tools/0.9.0-2ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted open-iscsi into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/open-iscsi/2.1.9-3ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Hello Viraniac, or anyone else affected,

Accepted dracut into noble-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dracut/060+5-1ubuntu3.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-noble to verification-done-noble. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-noble. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Chris Halse Rogers (raof)]

Ok. Everything but the cloud-initrafms-tools looks OK (you're still welcome to upload something along the lines of https://bugs.launchpad.net/ubuntu/+source/dracut/+bug/2065180/comments/44).

For cloud-initramfs-tools it seems like combining the silent-failure on lack of intel-aesni with the rest might be an important behaviour difference? Failing to add the other modules to the initramfs will result in an unbootable system, right? I'm not sure under what circumstances those modules could fail to exist, but if they don't exist, we should fail to create an initramfs?

[Benjamin Drung (bdrung)]

The proposal in comment #2065180-44 is interesting. That would address user hooks. But what about hooks shipped by packages outside the Ubuntu archive?

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted initramfs-tools (0.142ubuntu25.2) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

cryptsetup/unknown (s390x)
initramfs-tools/0.142ubuntu25.2 (s390x)
initramfs-tools/unknown (ppc64el)
kdump-tools/unknown (ppc64el)
mandos/unknown (ppc64el)
multipath-tools/0.9.4-5ubuntu8 (s390x)
multipath-tools/unknown (ppc64el)
zfs-linux/unknown (ppc64el, s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#initramfs-tools

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted dracut (060+5-1ubuntu3.2) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

clevis/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#dracut

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted open-iscsi (2.1.9-3ubuntu5.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

open-iscsi/2.1.9-3ubuntu5.1 (s390x)
resource-agents/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#open-iscsi

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted lvm2 (2.03.16-3ubuntu3.1) for noble have finished running.
The following regressions have been reported in tests triggered by the package:

cloud-utils/unknown (s390x)
cryptsetup/2:2.7.0-1ubuntu4 (s390x)
dm-writeboost/2.2.16-0.1ubuntu2 (arm64, ppc64el)
freedom-maker/0.33 (s390x)
multipath-tools/unknown (s390x)
rapiddisk/9.1.0-2build2 (arm64, armhf, ppc64el, s390x)
resource-agents/1:4.13.0-1ubuntu4 (s390x)
skopeo/unknown (s390x)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/noble/update_excuses.html#lvm2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Chris Halse Rogers (raof)]

> The proposal in comment #2065180-44 is interesting. That would address user hooks. But what about hooks shipped by packages outside the Ubuntu archive?

We don't support packages shipped outside the Ubuntu archive. If they break, they break.

But it's not *hugegly* likely that they'll break, either, so this would only impose a small chance of regression on people using an unsupported configuration.