Update to grub 2.04-1ubuntu44 breaks booting

[Guido]

Update to grub 2.04-1ubuntu44 causes system to fail to reboot properly.

(This is a continuation of Bug #1927853 .)

#lsb_release -a

LSB Version: core-9.20170808ubuntu1-noarch:printing-9.20170808ubuntu1-noarch:security-9.20170808ubuntu1-noarch
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04
Codename: bionic

Automatic software updater updated grub with target grub 2.04-1ubuntu44

#cat /var/log/apt/history.log
Start-Date: 2021-05-08 18:11:35
Commandline: aptdaemon role='role-commit-packages' sender=':1.62'
Install: grub-efi-amd64-signed:amd64 (1.167~18.04.1+2.04-1ubuntu44, automatic)
Upgrade: update-notifier-common:amd64 (3.192.1.9, 3.192.1.10), libsystemd0:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), libsystemd0:i386 (237-3ubuntu10.46, 237-3ubuntu10.47), grub-common:amd64 (2.02-2ubuntu8.21, 2.02-2ubuntu8.23), librbd1:amd64 (12.2.13-0ubuntu0.18.04.6, 12.2.13-0ubuntu0.18.04.7), grub2-common:amd64 (2.02-2ubuntu8.21, 2.02-2ubuntu8.23), udev:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), grub-pc:amd64 (2.02-2ubuntu8.21, 2.02-2ubuntu8.23), libudev1:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), libudev1:i386 (237-3ubuntu10.46, 237-3ubuntu10.47), grub-pc-bin:amd64 (2.02-2ubuntu8.21, 2.02-2ubuntu8.23), grub-efi-amd64-bin:amd64 (2.02-2ubuntu8.21, 2.04-1ubuntu44), libudev-dev:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), systemd-sysv:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), libpam-systemd:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), systemd:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), libnss-systemd:amd64 (237-3ubuntu10.46, 237-3ubuntu10.47), update-notifier:amd64 (3.192.1.9, 3.192.1.10), librados2:amd64 (12.2.13-0ubuntu0.18.04.6, 12.2.13-0ubuntu0.18.04.7)
End-Date: 2021-05-08 18:14:04

After subsequent shutdown, system fails to reboot correctly, leaving me at grub> prompt.

System is UEFI
HD is partitioned with ESP partition and encrypted partition.
Encrypted partition contains lvm volumes (lvm-boot, lvm-root, lvm-home, etc)

using grub> ls to find encrypted partition and performing grub> cryptomount makes lvm volumes accessible. Setting linux and initrd to appropriate files in lvm-boot allows subsequent successful boot.

Upgrade procedure appears to have lost previous configuration.

Looking at the /boot/efi/EFI/ubuntu/grub.cfg I see

#cat /boot/efi/EFI/ubuntu/grub.cfg
search.fs_uuid xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx root lvmid/yyyyyy-yyyy-yyyy-yyyy-yyyy-yyyy-yyyyyy/zzzzzz-zzzz-zzzz-zzzz-zzzz-zzzz-zzzzzz
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

The various UUIDs are all correct for my setup (UEFI. Clear-text ESP. Encrypted partition containing LVM volumes, with separate volumes for boot, root, home, var...)

The search.fs_uuid will fail if the key for the encrypted partition has not been given and the encrypted partition opened via cryptomount first, so it looks like update-grub is missing something it didn't before, which, I guess, has something to do with the transition from grub 2.02 to grub 2.04.

Running update-grub does not fix the problem.

(Bug previously submitted without apport)

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: grub-efi-amd64-bin 2.04-1ubuntu44
Uname: Linux 5.12.2-051202-generic x86_64
ApportVersion: 2.20.9-0ubuntu7.23
Architecture: amd64
Date: Sun May 9 15:27:07 2021
InstallationDate: Installed on 2017-09-05 (1342 days ago)
InstallationMedia: Lubuntu 16.04.3 LTS "Xenial Xerus" - Release amd64 (20170801)
SourcePackage: grub2-unsigned
UpgradeStatus: Upgraded to bionic on 2018-11-02 (918 days ago)

[Guido (guido-q)]

After another automatic software update of grub:

Extract from /var/log/apt/history.log

Start-Date: 2021-06-20 23:13:33
Commandline: aptdaemon role='role-commit-packages' sender=':1.135'
Upgrade: ..., grub-efi-amd64-signed:amd64 (1.167~18.04.3+2.04-1ubuntu44.1, 1.167~18.04.5+2.04-1ubuntu44.1.2), ...
End-Date: 2021-06-20 23:16:23

The grub,cfg on the EFI System Partition at /boot/efi/EFI/Ubuntu/grub.cfg was replaced with a new version.

I had edited the old version to include a necessary 'cryptomount' as the first line

Previous version
cryptomount -u <partition uuid without hyphens>
search.fs_uuid <filesystem uuid of /boot filesystem, with hyphens> root lvmid/<lvmid of volume group contining /boot filesystem>
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

This was replaced by a grub.cfg that lacked the first line
search.fs_uuid <filesystem uuid of /boot filesystem, with hyphens> root lvmid/<lvmid of volume group contining /boot filesystem>
set prefix=($root)'/grub'
configfile $prefix/grub.cfg

This obviously prevents successful booting.

This did not happen on previous upgrades before this bug was raised, so a breaking change* has occurred to the scripts that both generate a working grub.cfg and place it in the relevant place. Anyone who doesn't know their way around the grub command line and understand the use of cryptomount and lvm would have a very difficult time resolving the problem.

The quick resolution is
1) At the grub> prompt enter 'cryptomount -a', and provide the necessary passphrase
2) Point linux at the correct place: linux (lvm/<lvm volume that contains the boot filesystem>>)/<path to vmlinux>
3) Point initrd at the correct place: initrd (lvm/<volume group that contains the boot filesystem)/<path to initrd>
4) boot
5) Edit the new grub.cfg or replace with the old one.

*The breaking change appears to be that the script no longer correctly handles the encrypted partition.

[Julian Andres Klode (juliank)]

Did you set GRUB_ENABLE_CRYPTODISK? Please attach your /etc/default/grub.

Encrypted /boot is not really supported, the installer does not create it, and no testing is performed on it. You have to hack around things and add GRUB_ENABLE_CRYPTODISK to /etc/default/grub, so it's very hard to setup.

Last time I tried encrypted /boot, it would prompt me for password, but did not actually take any input.

[Julian Andres Klode (juliank)]

I forgot to mark this Incomplete but it would be expired now, so setting to Invalid directly.