initrd issue with hpsa kernel module drivers for HPE disk array controller

[Gustavo Sanchez]

======= Context
MAAS 3.3
Ubuntu Core {18, 20, 22 (Kernel 5.15 x86_64)}
Machine - HPE ProLiant DL360 Gen9
Disk Controller - HPE Smart Array P440ar Controller - Firmware 7.00

======= Description
Deploying Ubuntu Server 22.04 LTS (Kernel 5.15 x86_64) through MAAS on the HPE machine works as expected.
Deploying Ubuntu Core through MAAS on Supermicro machines works as expected.

Now, deploying Ubuntu Core on the HPE machine fails while booting trying to mount initial filesystems.

======= Logs
--- On Ubuntu Server checking the kernel module used for the disk array
ubuntu server # lspci -vvv
03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
    DeviceName: Embedded RAID 1
    Subsystem: Hewlett-Packard Company P440ar
# [..]
    Kernel driver in use: hpsa
    Kernel modules: hpsa

--- Logs from booting Ubuntu Core on HPE server
Failure log -> https://drive.google.com/file/d/1n0OMqVLdovO5ArbwxRojsfmRrrSvgFzd/view?usp=sharing
Full boot logs -> https://drive.google.com/file/d/1Zo3JsDjndzQs5oS5F8vgr6xiL0bg8GIA/view?usp=sharing

[Gustavo Sanchez (gustavosr98)]

Subscribing as "field-critical" as it is blocking a Field Engineering customer project.

[Ondrej Kubik (ondrak)]

@gustavo, you have more or less answered to yourself here :)
I double-checked, there is no "hpsa.ko" in the initrd, which means you have no storage to start with, that's the reason your "mount initial filesystems" is failing.
I had experienced an almost identical problem in the past with emmc on x86 when emmc module was missing in the initrd and boot was failing in the same way.

We can build a custom kernel snap with "hpsa.ko" included in the initrd to confirm this.

As a proper resolution, you would need to request hpsa.ko(it will also pull in scsi_transport_sas.ko) being added to the initrd from Ubuntu Core team. Size increase should be negligible (<100kB).