maas/pxe cannot boot with nvme device (ovmf works)

[Guilherme G. Piccoli]

MAAS relies on PXE to control its nodes (be it a bare-metal host or virtual machine); if machine is ready to get deployed (state ready), MAAS will PXE-boot the machine/guest to distro installer. After the installation, MAAS still PXE-boot the guest, but instructs it to sanboot local disks.

This bug report is related to PXE-booting a qemu guest to sanboot *with NVMe device* (I've tested the emulated one, I can try a PCI-PT too). It fails on seaboot, but works in OVMF. There's no PXE command to boot nvme (or at least I didn't find), but what OVMF seems to do is to fail PXE boot and fallback to local disks, be it SCSI/Virtio-blk or NVMe, and it has the desired effect (MAAS succeeds the deployment).
On seabios, the PXE-boot-to-local fails and the guest is stuck On older seabios (1.10.x), for some reason a PXE boot happens to the commission kernel (a MAAS thing) instead of getting the guest stuck.

The FW logs (and VNC screenshots) are attached - the versions I've used are:

# cat /etc/os-release |grep -i pretty
PRETTY_NAME="Ubuntu 18.04.4 LTS"

# dpkg -l | grep "qemu-kvm\|seabios\|ovmf\|libvirt0\|ipxe-qemu " | tr -s \ | cut -f1-3 -d \
ii ipxe-qemu 1.0.0+git-20190109.133f4c4-0ubuntu3
ii libvirt0:amd64 5.4.0-0ubuntu5.2~cloud0
ii ovmf 0~20191122.bd85bf54-2ubuntu3
ii qemu-kvm 1:4.0+dfsg-0ubuntu9.4~cloud0
ii seabios 1.13.0-1ubuntu1
(I'm using ipxe, seabios and ovmf from Focal, in order to get tests with more updated versions).

Next steps would be to test PCI-PT NVMe boot and upstream qemu+seabios+ipxe on Focal, to determine if the issue is present upstream.

[Guilherme G. Piccoli (gpiccoli)]

This could well be a MAAS bug (or even iPXE), as I'm not really sure in which component the flaw is. But given that MAAS+OVMF works, and I've confirmed both seabios and OVMF are using the same ipxe ROM, I believe this is a seabios issue.

MAAS PXE message sent to deployed guests is like below:

$ curl tftp://localhost/ipxe.cfg-52:54:00:ac:e9:7e
#!ipxe
# Boot the locally attached hard drives. On failure try the next locally
# attached hard drive. Fail on the last hard drive, if booting fails.
sanboot --no-describe --drive 0x80 ||
sanboot --no-describe --drive 0x81 ||
sanboot --no-describe --drive 0x82 ||
sanboot --no-describe --drive 0x83

This command was executed inside MAAS VM (the MAC address is for the nvme guest).

[Guilherme G. Piccoli (gpiccoli)]

This is the VNC screenshot of a failing boot with seabios 1.10 - it eventually booted the commission kernel via iPXE. If using 1.13, the screen is quite similar, but it gets stuck on "Booting from SAN device 0x81".

[Guilherme G. Piccoli (gpiccoli)]

After building Focal OVMF with (verbose) debug enabled, got this log.