virt-install of focal-server-cloudimg-amd64.img panics on first boot, breaking cloud-init

[David Baird]

Various Focal cloudimg builds panic on first time ever booting with "Cannot open root device" when booting with virt-install (libvirt). After panicing, it reboots, and the second boot succeeds (using initrd on the second try). However, it needs to succeed on the first try without a panic. This breaks the cloud-init NoCloud process which expects the first time booting to succeed.

E.g.
https://cloud-images.ubuntu.com/focal/20200418/focal-server-cloudimg-amd64.img

Here is the failed output:

[ 0.000000] Linux version 5.4.0-24-generic (buildd@lgw01-amd64-017) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #28-Ubuntu SMP Thu Apr 9 22:16:42 UTC 2020 (Ubuntu 5.4.0-24.28-generic 5.4.30)
[ 0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.4.0-24-generic root=PARTUUID=8a7c6798-57bf-4624-80a0-b896a73939f3 ro console=tty1 console=ttyS0 panic=-1
...
[ 1.043162] VFS: Cannot open root device "PARTUUID=8a7c6798-57bf-4624-80a0-b896a73939f3" or unknown-block(0,0): error -6
[ 1.045238] Please append a correct "root=" boot option; here are the available partitions:
[ 1.046921] 0b00 1048575 sr0
[ 1.046922] driver: sr
[ 1.048215] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 1.049795] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 5.4.0-24-generic #28-Ubuntu
[ 1.051189] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
[ 1.052904] Call Trace:
[ 1.053459] dump_stack+0x6d/0x9a
[ 1.054158] panic+0x101/0x2e3
[ 1.054812] mount_block_root+0x23f/0x2e8
[ 1.055645] mount_root+0x38/0x3a
[ 1.056319] prepare_namespace+0x13f/0x194
[ 1.057236] kernel_init_freeable+0x231/0x255
[ 1.058131] ? rest_init+0xb0/0xb0
[ 1.058834] kernel_init+0xe/0x100
[ 1.059535] ret_from_fork+0x35/0x40
[ 1.060327] Kernel Offset: 0x1f800000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)

When booted, the second time, it works!

The issue appears to be that cloudimg's kernel doesn't come with the necessary drivers pre-compiled into the kernel on a KVM/QEMU system. And, combined with a recent (2019) change to /boot/grub/grub.cfg which doesn't load initrd on the first boot, this ensures that the first time booting will fail.

Relevant section of /boot/grub/grub.cfg:

menuentry 'Ubuntu' --class ubuntu --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-aca11948-bdc4-4726-9aca-ead9dcce6a5e' {
        recordfail
        load_video
        gfxmode $linux_gfx_mode
        insmod gzio
        if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
        insmod part_gpt
        insmod ext2
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root  aca11948-bdc4-4726-9aca-ead9dcce6a5e
        else
          search --no-floppy --fs-uuid --set=root aca11948-bdc4-4726-9aca-ead9dcce6a5e
        fi
        if [ "${initrdfail}" = 1 ]; then
          linux /boot/vmlinuz-5.4.0-24-generic root=LABEL=cloudimg-rootfs ro  console=tty1 console=ttyS0
          initrd        /boot/initrd.img-5.4.0-24-generic
        else
          linux /boot/vmlinuz-5.4.0-24-generic root=LABEL=cloudimg-rootfs ro  console=tty1 console=ttyS0 panic=-1
        fi
        initrdfail
}

[David Baird (dhbaird)]

Discussing here: https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1870189