grub-install failure does not fail package upgrade (and does not roll back to matching modules)

[Robert C Jennings]

[SRU Justification]
SRU regression triggered by latent misconfiguration of grub on the host system, leading to a failure to boot due to ABI mismatch between /boot/grub/i386-pc which is updated, and the grub image in the MBR which is not

[Test case]
1. Boot a system in BIOS mode.
2. Run sudo sed -i -e'/Name: grub-pc\/install_devices/,/Name/ s,Value: .*,Value: /dev/non-existent,' /var/cache/debconf/config.dat
3. Enable -proposed in apt sources.list.
4. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-pc
5. Reboot
6. Confirm that the system boots successfully
7. Boot a system in UEFI mode.
8. Enable -proposed in apt sources.list.
9. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-efi-amd64-signed
10. Confirm that the contents of /boot/efi/EFI/ubuntu have been updated with the current timestamp.

[Original description]
Failure to install new grub core to the specified device does not correctly prevent upgrade to incompatible modules (LP: #1889509)

$ sudo debconf-get-selections |grep sda
grub-pc grub-pc/install_devices_disks_changed multiselect /dev/sda
grub-pc grub-pc/install_devices multiselect /dev/sda

$ mount|grep nvme
/dev/nvme0n1p1 on / type ext4 (rw,relatime,discard,data=ordered)

$ ls /dev/sda
ls: cannot access '/dev/sda': No such file or directory

$ sudo env DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
...
Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 grub-pc-bin amd64 2.02~beta2-36ubuntu3.26 [891 kB]
...
Installing for i386-pc platform.
grub-install: error: cannot find a GRUB drive for /dev/sda. Check your device.map.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-1111-aws
Found linux image: /boot/vmlinuz-4.4.0-1109-aws
Found initrd image: /boot/initrd.img-4.4.0-1109-aws
done
...

# update-grub failed during the install but the return code is 0
$ echo $?
0

# The package is installed without apparent error, but the instance will fail to reboot (LP: #1889509)
$ dpkg -l|grep grub-pc
ii grub-pc 2.02~beta2-36ubuntu3.26 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.02~beta2-36ubuntu3.26 amd64 GRand Unified Bootloader, version 2 (PC/BIOS binaries)

# If I reboot it will fail to boot:
Booting from Hard Disk 0...
error: symbol `grub_calloc' not found.
Entering rescue mode...
grub rescue> _

---

Xenial in AWS (us-west-2 ami-060d1be0dd4526759 built on 20200611)
The debconf for grub was not set to the correct device when cloud-init first ran (LP: #1877491) or when the fix for that was applied (LP: #1889555)
The fact that grub-install fails during the upgrade but does not fail the package install (and cause a rollback) means that how we have a mismatch between grub core and modules which breaks boot (LP: #1889509).

[Robert C Jennings (rcj)]

I'm attaching a full log from the recreate that shows additional details

[Dimitri John Ledkov (xnox)]

Hm, I wonder if in postinst, in grub-pc case,

                else
                  break # noninteractive
                fi

Should actually be

exit 1, like it is in the UPGRADE_FROM_GRUB_LEGACY case.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert C Jennings (rcj)]

My recreate is with an AWS EC2 instance with an NVMe root device as that is one known way to trigger an error in cloud-init cc_grub_dpkg so that it does not write a valid debconf for grub-pc install_devices, however this is broader than NVMe root with cloud-init.

[Patricia Gagnon-Renaud (pgrenaud-exolnet)]

We had this problem yesterday on only one of two Ubuntu 18.04 instance on AWS. We had to rollback to a previous snapshot to make the instance bootable again.

So we did a quick test with these two AMI:

ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408 - ami-0edd51cc29813e254
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 - ami-0d0eaed20348a3389

We deployed one fresh instance for each of those AMI and ran the same update process that we usually do (we the apt module from ansible to perform the updates). After the reboot, both instances had the issue (being stuck at "error: symbol `grub_calloc' not found").

Then, we did the same thing, but ran the apt upgrade manually on both instances and selected the default option on every prompt that shows up. After the reboot, both instances were normally accessible, without any sign of the issue.

Hope that this information can be useful!

[Dimitri John Ledkov (xnox)]

Installing for i386-pc platform.
grub-install: error: cannot find a GRUB drive for /dev/sda. Check your device.map.
dpkg: error processing package grub-pc (--configure):
 subprocess installed post-installation script returned error exit status 1
...
Errors were encountered while processing:
 grub-pc
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Brian Murray (brian-murray)]

Hello Robert, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

An upload of grub2 to bionic-proposed has been rejected from the upload queue for the following reason: "noise in debdiff, will repload".

[Steve Langasek (vorlon)]

Hello Robert, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Steve Langasek (vorlon)]

An upload of grub2 to xenial-proposed has been rejected from the upload queue for the following reason: "cruft in debdiff".

[Steve Langasek (vorlon)]

Hello Robert, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Dariusz Gadomski (dgadomski)]

bionic-verification:
Tested with version 1.93.19+2.02-2ubuntu8.17 from -proposed using the test case in the description.

Boots successfully in BIOS mode.
Timestamps were updated in EFI mode.

[Dariusz Gadomski (dgadomski)]

focal verification:
Tested with version 1.142.4+2.04-1ubuntu26.2 from -proposed using the above test case.

Boots successfully in BIOS mode.
Timestamps updated in EFI mode.

[Dariusz Gadomski (dgadomski)]

I have also attempted to verify xenial (using version 1.66.27+2.02_beta2-36ubuntu3.27), however grub-efi-amd64-signed_1.66.27+2.02_beta2-36ubuntu3.27 still seems to be unavailable in -proposed (http://archive.ubuntu.com).

I have manually downloaded it and tested from here [1], however I'll postpone the verification until it's available.

[1] https://launchpad.net/ubuntu/+archive/primary/+files/grub-efi-amd64-signed_1.66.27+2.02~beta2-36ubuntu3.27_amd64.deb

[Dariusz Gadomski (dgadomski)]

xenial verification:
Tested with 1.66.27+2.02_beta2-36ubuntu3.27 from -proposed.

Boots successfully in BIOS mode.
Timestamps updated in EFI mode.

[Dariusz Gadomski (dgadomski)]

I have run some additional tests on bionic and focal desktop VMs with lvm (and lvm+luks) - no boot issues were observed with the -proposed builds.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu26.2

---------------
grub2 (2.04-1ubuntu26.2) focal; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 17:34:25 -0700

[Steve Langasek (vorlon)]

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02-2ubuntu8.17

---------------
grub2 (2.02-2ubuntu8.17) bionic; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 18:49:49 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.02~beta2-36ubuntu3.27

---------------
grub2 (2.02~beta2-36ubuntu3.27) xenial; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 21:27:00 -0700

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu26.2

---------------
grub2 (2.04-1ubuntu26.2) focal; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 17:34:25 -0700

[halfgaar (wiebe-halfgaar)]

I had an amazon EC2 Ubuntu 18.04 instance fail to boot. Grub was updated to the fixed version on aug 3 (2.02-2ubuntu8.17). A fix for this bug that prevents this in the future may have been released, but the broken state the machine is in, is not recovered.

Amazon AWS is especially tricky, because you don't have a VGA console to see the error in. Even tier 1 support only has virtual console access, which is not there yet.

[Seth Arnold (seth-arnold)]

Hello halfgaar, our knowledge base article includes some instructions for Amazon ec2 instances on how to recover from this situation:

https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/GRUB2SecureBootBypass#Cloud_instances_.28e.g._AWS_EC2.29

If the virtual console support provides a mechanism to supply boot media, you could boot into a rescue / live environment, chroot into your existing installation, and run dpkg-reconfigure grub-pc. If the virtual console support doesn't provide the ability to provide temporary boot media, then the longer, more complicated, instructions will be required.

Thanks

[Juergen (juergen-schwarz)]

To see the error on the VGA console of an AWS EC2 instance:

apt-get install netpbm gocr
aws --output=text ec2 get-console-screenshot --instance-id $InstanceId --query ImageData | base64 -d | jpegtopnm | gocr -

[Bill McGonigle (bill-launchpad-net)]

Fast forwarding eight months, this still breaks a fully-updated xubuntu LTS 16 to LTS 18 release upgrade, at least. Does this need to be pushed to LTS 16 (or 18 if the new package is run)?

Mounting the disks on the host (error was in a VM guest), chrooting, and running grub-install /dev/sda fixes, so this doesn't seem like a complex disk layout issue. I'm about to see if fully-updated LTS 18 to LTS 20 also breaks ... yep 'grub_file_filters' is the symbol this time. Same fix.

Is it correct that do-release-upgrade has been broken in this way for most of the past year?

[Steve Langasek (vorlon)]

On Thu, May 13, 2021 at 09:21:22PM -0000, Bill McGonigle wrote:
> Fast forwarding eight months, this still breaks a fully-updated xubuntu
> LTS 16 to LTS 18 release upgrade

Because you have a misconfigured grub package on your system which is not
writing to the correct MBR. That is impossible for us to fix from the
package, as the bug is in the package configuration and there is no way to
programmatically determine the correct target for grub-install.

> Mounting the disks on the host (error was in a VM guest), chrooting, and
> running grub-install /dev/sda fixes, so this doesn't seem like a complex
> disk layout issue. I'm about to see if fully-updated LTS 18 to LTS 20
> also breaks ... yep 'grub_file_filters' is the symbol this time. Same
> fix.

If 'grub-install /dev/sda' fixes it for you, then you should fix your grub
config to point to /dev/sda using 'sudo dpkg-reconfigure grub-pc'.