grub-install failure does not fail package upgrade (and does not roll back to matching modules)

Bug #1889556 reported by Robert C Jennings on 2020-07-30
412
This bug affects 47 people
Affects Status Importance Assigned to Milestone
grub2 (Ubuntu)
Status tracked in Groovy
Xenial
Undecided
Unassigned
Bionic
Undecided
Unassigned
Focal
Undecided
Unassigned
Groovy
Undecided
Unassigned

Bug Description

[SRU Justification]
SRU regression triggered by latent misconfiguration of grub on the host system, leading to a failure to boot due to ABI mismatch between /boot/grub/i386-pc which is updated, and the grub image in the MBR which is not

[Test case]
1. Boot a system in BIOS mode.
2. Run sudo sed -i -e'/Name: grub-pc\/install_devices/,/Name/ s,Value: .*,Value: /dev/non-existent,' /var/cache/debconf/config.dat
3. Enable -proposed in apt sources.list.
4. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-pc
5. Reboot
6. Confirm that the system boots successfully
7. Boot a system in UEFI mode.
8. Enable -proposed in apt sources.list.
9. Run sudo env DEBIAN_FRONTEND=noninteractive apt install grub-efi-amd64-signed
10. Confirm that the contents of /boot/efi/EFI/ubuntu have been updated with the current timestamp.

[Original description]
Failure to install new grub core to the specified device does not correctly prevent upgrade to incompatible modules (LP: #1889509)

$ sudo debconf-get-selections |grep sda
grub-pc grub-pc/install_devices_disks_changed multiselect /dev/sda
grub-pc grub-pc/install_devices multiselect /dev/sda

$ mount|grep nvme
/dev/nvme0n1p1 on / type ext4 (rw,relatime,discard,data=ordered)

$ ls /dev/sda
ls: cannot access '/dev/sda': No such file or directory

$ sudo env DEBIAN_FRONTEND=noninteractive apt full-upgrade -y
...
Get:10 http://us-west-2.ec2.archive.ubuntu.com/ubuntu xenial-updates/main amd64 grub-pc-bin amd64 2.02~beta2-36ubuntu3.26 [891 kB]
...
Installing for i386-pc platform.
grub-install: error: cannot find a GRUB drive for /dev/sda. Check your device.map.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.4.0-1111-aws
Found linux image: /boot/vmlinuz-4.4.0-1109-aws
Found initrd image: /boot/initrd.img-4.4.0-1109-aws
done
...

# update-grub failed during the install but the return code is 0
$ echo $?
0

# The package is installed without apparent error, but the instance will fail to reboot (LP: #1889509)
$ dpkg -l|grep grub-pc
ii grub-pc 2.02~beta2-36ubuntu3.26 amd64 GRand Unified Bootloader, version 2 (PC/BIOS version)
ii grub-pc-bin 2.02~beta2-36ubuntu3.26 amd64 GRand Unified Bootloader, version 2 (PC/BIOS binaries)

# If I reboot it will fail to boot:
Booting from Hard Disk 0...
error: symbol `grub_calloc' not found.
Entering rescue mode...
grub rescue> _

---

Xenial in AWS (us-west-2 ami-060d1be0dd4526759 built on 20200611)
The debconf for grub was not set to the correct device when cloud-init first ran (LP: #1877491) or when the fix for that was applied (LP: #1889555)
The fact that grub-install fails during the upgrade but does not fail the package install (and cause a rollback) means that how we have a mismatch between grub core and modules which breaks boot (LP: #1889509).

Related branches

Robert C Jennings (rcj) wrote :

I'm attaching a full log from the recreate that shows additional details

description: updated
tags: added: regression-release
tags: added: regression-update
information type: Public → Public Security
tags: added: regression-security
Dimitri John Ledkov (xnox) wrote :

Hm, I wonder if in postinst, in grub-pc case,

                else
                  break # noninteractive
                fi

Should actually be

exit 1, like it is in the UPGRADE_FROM_GRUB_LEGACY case.

Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in grub2 (Ubuntu Bionic):
status: New → Confirmed
Changed in grub2 (Ubuntu Focal):
status: New → Confirmed
Changed in grub2 (Ubuntu Xenial):
status: New → Confirmed
Changed in grub2 (Ubuntu):
status: New → Confirmed
Robert C Jennings (rcj) wrote :

My recreate is with an AWS EC2 instance with an NVMe root device as that is one known way to trigger an error in cloud-init cc_grub_dpkg so that it does not write a valid debconf for grub-pc install_devices, however this is broader than NVMe root with cloud-init.

We had this problem yesterday on only one of two Ubuntu 18.04 instance on AWS. We had to rollback to a previous snapshot to make the instance bootable again.

So we did a quick test with these two AMI:

ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20200408 - ami-0edd51cc29813e254
ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-20190722.1 - ami-0d0eaed20348a3389

We deployed one fresh instance for each of those AMI and ran the same update process that we usually do (we the apt module from ansible to perform the updates). After the reboot, both instances had the issue (being stuck at "error: symbol `grub_calloc' not found").

Then, we did the same thing, but ran the apt upgrade manually on both instances and selected the default option on every prompt that shows up. After the reboot, both instances were normally accessible, without any sign of the issue.

Hope that this information can be useful!

Steve Langasek (vorlon) on 2020-07-30
tags: removed: regression-release
Dimitri John Ledkov (xnox) wrote :

Installing for i386-pc platform.
grub-install: error: cannot find a GRUB drive for /dev/sda. Check your device.map.
dpkg: error processing package grub-pc (--configure):
 subprocess installed post-installation script returned error exit status 1
...
Errors were encountered while processing:
 grub-pc
E: Sub-process /usr/bin/dpkg returned an error code (1)

Dimitri John Ledkov (xnox) wrote :
Steve Langasek (vorlon) on 2020-07-31
description: updated

Hello Robert, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Focal):
status: Confirmed → Fix Committed
tags: added: verification-needed verification-needed-focal
Steve Langasek (vorlon) on 2020-07-31
description: updated

An upload of grub2 to bionic-proposed has been rejected from the upload queue for the following reason: "noise in debdiff, will repload".

Hello Robert, or anyone else affected,

Accepted grub2 into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02-2ubuntu8.17 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Bionic):
status: Confirmed → Fix Committed
tags: added: verification-needed-bionic

An upload of grub2 to xenial-proposed has been rejected from the upload queue for the following reason: "cruft in debdiff".

description: updated

Hello Robert, or anyone else affected,

Accepted grub2 into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.02~beta2-36ubuntu3.27 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in grub2 (Ubuntu Xenial):
status: Confirmed → Fix Committed
tags: added: verification-needed-xenial
Steve Langasek (vorlon) on 2020-07-31
description: updated
tags: added: sts
Dariusz Gadomski (dgadomski) wrote :

bionic-verification:
Tested with version 1.93.19+2.02-2ubuntu8.17 from -proposed using the test case in the description.

Boots successfully in BIOS mode.
Timestamps were updated in EFI mode.

tags: added: verification-done-bionic
removed: sts verification-needed-bionic
Dariusz Gadomski (dgadomski) wrote :

focal verification:
Tested with version 1.142.4+2.04-1ubuntu26.2 from -proposed using the above test case.

Boots successfully in BIOS mode.
Timestamps updated in EFI mode.

tags: added: verification-done-focal
removed: verification-needed-focal
Dariusz Gadomski (dgadomski) wrote :

I have also attempted to verify xenial (using version 1.66.27+2.02_beta2-36ubuntu3.27), however grub-efi-amd64-signed_1.66.27+2.02_beta2-36ubuntu3.27 still seems to be unavailable in -proposed (http://archive.ubuntu.com).

I have manually downloaded it and tested from here [1], however I'll postpone the verification until it's available.

[1] https://launchpad.net/ubuntu/+archive/primary/+files/grub-efi-amd64-signed_1.66.27+2.02~beta2-36ubuntu3.27_amd64.deb

Dariusz Gadomski (dgadomski) wrote :

xenial verification:
Tested with 1.66.27+2.02_beta2-36ubuntu3.27 from -proposed.

Boots successfully in BIOS mode.
Timestamps updated in EFI mode.

tags: added: verification-done-xenial
removed: verification-needed-xenial
tags: added: verification-done
removed: verification-needed
Dariusz Gadomski (dgadomski) wrote :

I have run some additional tests on bionic and focal desktop VMs with lvm (and lvm+luks) - no boot issues were observed with the -proposed builds.

Steve Langasek (vorlon) on 2020-07-31
Changed in grub2 (Ubuntu Groovy):
status: Confirmed → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.04-1ubuntu26.2

---------------
grub2 (2.04-1ubuntu26.2) focal; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 17:34:25 -0700

Changed in grub2 (Ubuntu Focal):
status: Fix Committed → Fix Released

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.02-2ubuntu8.17

---------------
grub2 (2.02-2ubuntu8.17) bionic; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 18:49:49 -0700

Changed in grub2 (Ubuntu Bionic):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.02~beta2-36ubuntu3.27

---------------
grub2 (2.02~beta2-36ubuntu3.27) xenial; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 21:27:00 -0700

Changed in grub2 (Ubuntu Xenial):
status: Fix Committed → Fix Released
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub2 - 2.04-1ubuntu26.2

---------------
grub2 (2.04-1ubuntu26.2) focal; urgency=medium

  * debian/postinst.in: Avoid calling grub-install on upgrade of the grub-pc
    package, since we cannot be certain that it will install to the correct
    disk and a grub-install failure will render the system unbootable.
    LP: #1889556.

 -- Steve Langasek <email address hidden> Thu, 30 Jul 2020 17:34:25 -0700

Changed in grub2 (Ubuntu Groovy):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public Security information  Edit
Everyone can see this security related information.

Duplicates of this bug

Other bug subscribers