unmkinitramfs is very slow on initrd from ubuntu 23.10+, solution proposed

[Adam Vodopjan]

Since 23.10 the initrd consists of 4 embedded cpio:
- amd microcode
- intel microcode
- firmware + kernel modules
- rest (compressed)

Previously it was:
- amd microcode
- intel microcode
- rest (compressed)

There is no archive size header for cpio, so unmkinitramfs has to parse headers of each entry in uncompressed archives.

> time unmkinitramfs 23.04/{initrd,unpack}

real 0m3.625s
user 0m1.595s
sys 0m3.467s

> time unmkinitramfs 23.10/{initrd,unpack}

real 0m15.982s
user 0m20.174s
sys 0m6.995s

> for dir in 23.04/unpack/early*; do printf '%-20s %s\n' "$dir" "$(find "$dir" | wc -l)"; done
23.04/unpack/early 5
23.04/unpack/early2 6

> for dir in 23.10/unpack/early*; do printf '%-20s %s\n' "$dir" "$(find "$dir" | wc -l)"; done
23.10/unpack/early 5
23.10/unpack/early2 6
23.10/unpack/early3 2874

Here https://github.com/slowpeek/unmkinitramfs-turbo I've made it a lot faster with two changes:
- bash instead of sh for bash's "read -N"
- xxd dump of initrd instead of raw binary data

I wonder, if the developers are interested in accepting my improvements into the code base?

> time unmkinitramfs-classic-turbo 23.10/{initrd,unpack}

real 0m2.209s
user 0m2.161s
sys 0m1.341s

> time unmkinitramfs-turbo 23.10/{initrd,unpack}
 early .. +
early2 .. +
early3 .. +
  main .. +

real 0m1.973s
user 0m2.094s
sys 0m1.244s

All tools in the repo, even the classic one, feature -s/--scan option. With it, the tools only dump offset + size for embedded archives, nothing is unpacked. It can be used to measure parsing speed.

In the repo there are three tools:
- unmkinitramfs-classic: the original tool
- unmkinitramfs-classic-turbo: the improved tool with minimal changes
- unmkinitramfs-turbo: the improved tool with many changes to my tastes, extra dependency on "file" and some additional features (-ss option, decompression with 7za)

[Benjamin Drung (bdrung)]

Thank you for taking the time to report this bug and helping to make Ubuntu better. Speed improvements are welcome. Please submit those changes (preferred in the smallest possible chunks).

Maybe we should rewrite splitinitramfs (or the complete script) in C.

[Benjamin Drung (bdrung)]

Your bug report triggered me to write a command that can list or extract the content of the initramfs in Rust, since Rust would be a good language for that task. I have implemented the listing of uncompressed cpio archives (roughly 120 lines of code). Now I have to add the decompression support. Then I will share my results.

[Benjamin Drung (bdrung)]

The result is promising. I tested on my Ryzen 5700G desktop with the mantic initrd:

```
$ ls /boot/initrd.img-6.5.0-26-generic -l
-rw-r--r-- 1 root root 106445830 Mär 18 18:30 /boot/initrd.img-6.5.0-26-generic
$ time lsinitramfs /boot/initrd.img-6.5.0-26-generic > /dev/null

real 0m13,960s
user 0m17,944s
sys 0m5,098s
$ time ./unmkinitramfs-turbo -l /boot/initrd.img > /dev/null

real 0m1,824s
user 0m2,099s
sys 0m0,915s
$ time cargo run > /dev/null

real 0m0,190s
user 0m0,037s
sys 0m0,147s
$ cargo build --release
$ time target/release/init-cpio > /dev/null

real 0m0,155s
user 0m0,027s
sys 0m0,121s
```

Cutting the time to list the content from roughly 14 second down to 160 ms (1% of the time or factor of 90). Compared to your unmkinitramfs-turbo it is still over 11 times faster.

Okay, I still have to add some consistency check (check the magic), but I can still optimize the code. But before doing that, I need to clean up the code to publish it somewhere.

[Adam Vodopjan (grozzly)]

> I have implemented the listing of uncompressed cpio archives (roughly 120 lines of code). Now I have to add the decompression support

Finding offsets of the embedded archives is the only time consuming part of the whole script. Mby it is better to create a separate tool, say "scaninitramfs", to only list offsets and size of embedded archives, just like my tools' "-s" option does, and use it in the original script?

Scan output can be of interest beyond unmkinitramfs. For example, if you need to know the compression used in some initrd. With a scan tool you can quickly get the offset of the compressed cpio and "dd .. | file .."

[Benjamin Drung (bdrung)]

I could easily add an examine option to show what the initramfs consists of, e.g.

$ init-cpio --examine /boot/initrd.img
0: uncompressed cpio
123: uncompressed cpio
456: uncompressed cpio
7890: zstd compressd cpio

[Benjamin Drung (bdrung)]

I published the first version of the Rust implementation: https://github.com/bdrung/icpio

icpio is 88 to 274 times faster than lsinitramfs. See README.md from icpio.