Can't set boot kernel in GRUB on focal image

[Ben Romer]

It appears that the 20.04 cloud image does not update-grub properly when GRUB_DEFAULT is changed.

As part of my livepatch testing process, I will install an older kernel release on a cloud image, and then set the image to boot the older kernel, which is how I noticed this problem. However, the bug is reproducible without installing anything.

Steps to reproduce:
1. create a new focal VM using 'uvt-kvm create test release=focal arch=amd64'.
2. SSH into the new VM using 'uvt-kvm ssh test'.
3. Change the default boot entry to boot the installed kernel using the advanced menu, by using this command:

  echo "GRUB_DEFAULT='Advanced options for Ubuntu>Ubuntu, with Linux $(uname -r)'" | sudo tee /etc/default/grub.d/99-override-kernel.cfg

4. Update the grub files using 'sudo update-grub'.
5. Reboot the VM using 'sudo reboot'.

WHAT I EXPECTED:
The VM should reboot into the specified kernel. This does work correctly on bionic and xenial VMs.

WHAT HAPPENED:
The VM fails to boot. Instead, it crashes in an infinite loop.

[ 1.451958] VFS: Cannot open root device "PARTUUID=f1fd35e8-b25d-4dea-8e33-06fc04e8fcdf" or unknown-block(0,0): error -6
[ 1.454532] Please append a correct "root=" boot option; here are the available partitions:
[ 1.456671] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 1.458372] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.4.0-39-generic #43-Ubuntu
[ 1.459935] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.13.0-1ubuntu1 04/01/2014
[ 1.461705] Call Trace:
[ 1.462312] dump_stack+0x6d/0x9a
[ 1.463064] panic+0x101/0x2e3
[ 1.463917] mount_block_root+0x23f/0x2e8
[ 1.464786] mount_root+0x38/0x3a
[ 1.465595] prepare_namespace+0x13f/0x194
[ 1.466503] kernel_init_freeable+0x231/0x255
[ 1.467448] ? rest_init+0xb0/0xb0
[ 1.468213] kernel_init+0xe/0x100
[ 1.469061] ret_from_fork+0x35/0x40
[ 1.471399] Kernel Offset: 0x2ce00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1.474038] ACPI MEMORY or I/O RESET_REG.

[Ben Romer (bromer)]

I've attached the crash that happens after the boot modification, and a copy of the dmesg from a successful boot without any modification.

[Robert C Jennings (rcj)]

Adding 'grub' project to verify and provide assistance.

[Steve Langasek (vorlon)]

grub is the obsolete grub1 source package. reassigning.

[Cody Shepherd (codyshepherd)]

Confirmed on Multipass using the 20200702 focal release image. Also captured a sample selection of virsh console output during the infinite boot loop: https://paste.ubuntu.com/p/dqqgqryXNV/

[Richard Anderson (richmbx)]

I noticed this happens when simply rebooting.

1. Import Focal ova image into ESXi/vSphere
2. Setup new virtual machine with little or no changes
3. Reboot via command line (shutdown -r now) or ctrl-alt-del
4. First boot after the initiated reboot results in a kernel panic (similar to above)
5. System reboots and boots into Ubuntu

I have this occurring on Focal OVA images with ESXI 6.5 and Fusion 11.5.7. This does not occur with Bionic or Groovy.