Ubuntu server 22.04.3 fails to boot with HWE kernel (initramfs issue / panic)

[Guilherme G. Piccoli]

While trying to install Ubuntu server 22.04.3 ISO in a virtual machine (qemu), I seem to be unable to boot successfully if choosing the HWE option in the GRUB menu.

What happens is that it fails boot with a kernel panic, caused due to `/init` problems - I've collected the dmesg, and the message observed is: "Failed to execute /init (error -2)".
Checking the kernel code, *seems* run_init_process() failed on kernel_execve() with -ENOENT.

The regular kernel boots fine (when I choose the first option on GRUB menu) - interestingly, by decompressing both `initrd` and `hwe-initrd`, seems `/init` is the *same* file, based on its checksum. Maybe something wrong with hwe-initrd file itself that makes kernel unhappy on decompressing, or maybe a problem in my system?

Thanks in advance for taking a look, lemme know if I can help in some way (collecting more data, for example).

I'll attach the full dmesg, but notice is easy to reproduce that by trying to kexec the pairs hwe-vmlinuz/hwe-initrd or even vmlinuz/hwe-initrd (I tried with the old kernel version just to be sure!)

[Dan Bungert (dbungert)]

Hi Guilherme, thanks for the bug report.

I tried to recreate this today and did not have success. Please provide comprehensive steps to reproduce to show this problem.

I don't think this is an overall HWE kernel problem as I was able to install a system with only the linux-generic-hwe-22.04 kernel present and it was working fine. A second test with both linux-generic-hwe-22.04 and linux-generic, which defaults to the HWE one, also seems ok.

[Guilherme G. Piccoli (gpiccoli)]

Interesting @dbungret, thanks for your test!

What ISO did you try? Mine is downloaded from https://releases.ubuntu.com/22.04.3/ubuntu-22.04.3-live-server-amd64.iso - the md5sum is: 2072007c1d5b504c9a4858240e28e669

The comprehensive (quick) test for this:

1) wget https://releases.ubuntu.com/22.04.3/ubuntu-22.04.3-live-server-amd64.iso

2) Boot a qemu VM with the attached XML (a suggestion, of course), adding the CD-ROM as boot device 1.

3) Boot such VM using the virt-manager, and select the HWE kernel.

Alternatively to step (2)-(3), if you already have some qemu VM, you could install kexec-tools and mount such image inside the VM, then (let's assume the ISO is mounted as /mnt): kexec -s -l /mnt/casper/hwe-vmlinuz --initrd /mnt/casper/hwe-initrd --command-line="loglevel=8 console=ttyS0". With the VM console opened, then run "kexec -e" - with this kexec trick, I reproduced it here consistently.

Thanks again, and let me know if I can help with anything (testing stuff, collecting logs, etc)!

P.S. Maybe worth mentioning some versions of packages I'm running in my host system (Ubuntu 20.04):

ii libvirt0:amd64 8.6.0-0ubuntu3.1~backport20.04.202212052147~ubuntu20.04.1
ii qemu-system-x86 1:7.0+dfsg-7ubuntu2.3~backport20.04.202304142049~ubuntu20.04.1
ii seabios 1.16.0-4~backport20.04.202211181046~ubuntu20.04.1

$ uname -srv
Linux 5.15.0-67-generic #74~20.04.1-Ubuntu SMP Wed Feb 22 14:52:34 UTC 2023

[Guilherme G. Piccoli (gpiccoli)]

Ah Dan, I forgot to respond to you!

"I don't think this is an overall HWE kernel problem " <- me neither, for sure! I'm booting this SAME VM with the HWE kernel heh

My theory/intuition says it's related with the hwe-initrd file, maybe during the kernel decompression something is not going fine, and then kernel reads /init in a wrong fashion, i.e., it's not reading the proper file!

Cheers