Failing to deploy Lunar (23.04) and Mantic (23.10)

[DUFOUR Olivier]

The MAAS images for Lunar and Mantic doesn't seem to be working currently.

When deploying servers on Lunar and Mantic, the servers are all failing to retrieve the squashfs because the Initrd used during the PXE boot phase, does not provide any module for network devices
When looking at the server from the initramfs shell it appears :
- no network interfaces are detected at all, only "lo" is visible
- there is no modules for Virtio and e1000e Intel network devices at least

Fails on :
* MAAS 3.3.4
* Network card : e1000e and virtio_pci
* Ubuntu releases :
  * Lunar (23.04)
  * Mantic (23.10)
* image repository : http://images.maas.io/ephemeral-v3/candidate/

The same machines are confirmed to work on :
* Focal (20.04)
* Jammy (22.04)
* Kinetic (22.10)

[Igor Brovtsin (igor-brovtsin)]

Hi ~odufourc!

Could you please also attach the JSON generated by MAAS at `http://<maas-ip:port>/MAAS/images-stream/streams/v1/maas:v2:download.json`? A quick check of the most recent Lunar initrd contents (https://images.maas.io/ephemeral-v3/candidate/lunar/amd64/20230524/ga-23.04/generic/boot-initrd) showed that both `virtio_dma_buf.ko` and `e1000.ko` are present there.

[DUFOUR Olivier (odufourc)]

Hello Igor,

I didn't know this URL to retrieve all the information of images. But that is certainly useful.
I'm attaching the JSON to this comment.
Indeed, after comparing both Jammy and Lunar boot-initrd file, I do notice the network modules are present but somehow the modules are not loaded during the boot on Lunar and Mantic.

When arriving on the initramfs shell, no interfaces are detected, but if I do a modprobe then the interfaces appear.
The behavior is the same with virtio_net or with e1000e

[DUFOUR Olivier (odufourc)]

I have tested with a physical machine, a Dell R330, and the following could be noted :
1) the server has 2 network card :
 * Broadcom BCM5720 (module tg3)
 * Mellanox Connectx-3 (module mlx4_core)
And none of them were detected initially. But if I run manually a modprobe with the related module, then all the interfaces are appearing.

2) On that server, the USB keyboard was not working either, so I'm guessing that USB related modules were not loaded (if not all kernel modules in fact)

I managed to do some quick tests through the serial console thus I'm attaching the complete boot log from the PXE/deployment phase.

[Igor Brovtsin (igor-brovtsin)]

It turns out that candidate stream `lunar` uses `Linux version 6.2.0-20-generic` that does not include a fix for LP:2016908 that causes such behaviour. Workaround mentioned there (including `apparmor=1` kernel cmdline) worked for me locally, please check if it helps. The issue will be resolved once the updated kernel reaches the candidate stream