Error: out of memory while booting after installing the linux-firmware in proposed

[Kevin Yeh]

I found that there are some machines in cert lab can't boot after installing the linux-frimware in proposed with 6.8.0-40 kernel.

After selecting booting with the 6.8.0-40 kernel, it shows error: out of memory on the screen then freeze.

There aren't any journal logs available, since they froze at very early stage.

Following are the machines got impacted for now
https://certification.canonical.com/hardware/202306-31695/
https://certification.canonical.com/hardware/202306-31696/

[Kai-Heng Feng (kaihengfeng)]

The initramfs became too large to be loaded, due to the new nvidia firmwares introduced by LP: #2076180.

[Juerg Haefliger (juergh)]

The size of the resulting initrd is not a firmware package problem but an initramfs-tools issue.

[Juerg Haefliger (juergh)]

Hm. The initrd with the .32 firmware package is only marginally bigger, so there might be something else at play here:
-rw-r--r-- 1 ubuntu ubuntu 39289006 Aug 21 08:20 initrd.img-5.15.0-119-generic-31
-rw-r--r-- 1 ubuntu ubuntu 39289089 Aug 21 08:21 initrd.img-5.15.0-119-generic-32

[Juerg Haefliger (juergh)]

I guess I should use an nvidia kernel. Duh...

-rw-r--r-- 1 ubuntu ubuntu 129591289 Aug 21 08:39 initrd.img-6.8.0-1012-nvidia-31
-rw-r--r-- 1 ubuntu ubuntu 318674124 Aug 21 08:38 initrd.img-6.8.0-1012-nvidia-32

Yikes.

[Juerg Haefliger (juergh)]

Seems to be the new gsp files, whatever they are:

136K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ad102/gsp
104K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/tu117/gsp
136K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ad103/gsp
112K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga100/gsp
136K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ad106/gsp
136K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ad107/gsp
37M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga107/gsp
37M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga106/gsp
136K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ad104/gsp
23M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/tu106/gsp
37M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga104/gsp
23M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/tu102/gsp
37M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga103/gsp
37M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/ga102/gsp
104K 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/tu116/gsp
23M 6.8.0-1012-nvidia-32/main/lib/firmware/nvidia/tu104/gsp

[Timo Aaltonen (tjaalton)]

the whole nvidia dir is only 63MB on the git tree, so maybe the initramfs tools are a bit shit

[Timo Aaltonen (tjaalton)]

arch had the same issue, but using different tools it seems

https://gitlab.archlinux.org/archlinux/mkinitcpio/mkinitcpio/-/issues/238

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Timo Aaltonen (tjaalton)]

and there's this

https://airlied.blogspot.com/2023/04/kernel-module-firmware-selection-for.html

[Juerg Haefliger (juergh)]

Ah yes, initramfs-tools doesn't honor symlinks and dups the binaries :-(

firmware dir of orig initrd:
406M firmware.orig/

firmware dir of de-duped initrd:
194M firmware.dedup/

We might want to consider shipping an initramfs-hook script with the firmware package that recreates the symlinks.

[Benjamin Drung (bdrung)]

Oh, no. About which affected Ubuntu versions are we talking about?

[Juerg Haefliger (juergh)]

> Ah yes, initramfs-tools doesn't honor symlinks and dups the binaries :-(

That's an incorrect statement. It does copy symlinks. It looks like the firmware package ships duped binaries.

lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ad102/gsp/gsp-535.113.01.bin -> ../../ga102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ad103/gsp/gsp-535.113.01.bin -> ../../ga102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ad104/gsp/gsp-535.113.01.bin -> ../../ga102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ad106/gsp/gsp-535.113.01.bin -> ../../ga102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ad107/gsp/gsp-535.113.01.bin -> ../../ga102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 ga100/gsp/gsp-535.113.01.bin -> ../../tu102/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 38061600 Aug 13 09:49 ga102/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 38061600 Aug 13 09:49 ga103/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 38061600 Aug 13 09:49 ga104/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 38061600 Aug 13 09:49 ga106/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 38061600 Aug 13 09:49 ga107/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 23750944 Aug 13 09:49 tu102/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 23750944 Aug 13 09:49 tu104/gsp/gsp-535.113.01.bin
-rw-r--r-- 1 ubuntu ubuntu 23750944 Aug 13 09:49 tu106/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 tu116/gsp/gsp-535.113.01.bin -> ../../tu102/gsp/gsp-535.113.01.bin
lrwxrwxrwx 1 ubuntu ubuntu 34 Aug 21 15:56 tu117/gsp/gsp-535.113.01.bin -> ../../tu102/gsp/gsp-535.113.01.bin

[Juerg Haefliger (juergh)]

In Noble we actually explicitly de-dup firmware binaries during package build. I shall look into backporting that to Jammy.