(Thinkpad T470) Boot loop after update

[Moritz Baumann]

After upgrading shim-signed to 1.33.1~17.10.1 (and installing the grub updates that were published at the same time), the system is stuck in a boot loop. I've had to downgrade shim and grub to their release versions to get the system to boot again.

The following message gets displayed before the system resets each time:
> System BootOrder not found. Initializing defaults.
> Creating boot entry "Boot0000" with label "ubuntu" for file "\EFI\ubuntu\shim64.efi"
>
> Reset system

A similar message, without the last line, was already displayed on successful boots before the update.

According to the information in bug #1747889, the Thinkpad X1 Carbon might be affected as well.

ProblemType: Bug
DistroRelease: Ubuntu 17.10
Package: shim 0.9+1474479173.6c180c6-1ubuntu1
ProcVersionSignature: Ubuntu 4.13.0-32.35-generic 4.13.13
Uname: Linux 4.13.0-32-generic x86_64
.proc.sys.kernel.moksbstate_disabled: Error: [Errno 2] No such file or directory: '/proc/sys/kernel/moksbstate_disabled'
ApportVersion: 2.20.7-0ubuntu3.7
Architecture: amd64
CurrentDesktop: ubuntu:GNOME
Date: Mon Feb 19 11:10:34 2018
Dependencies:

EFITables:
 Feb 19 10:08:21 mdbauman kernel: efi: EFI v2.50 by Lenovo
 Feb 19 10:08:21 mdbauman kernel: efi: SMBIOS=0xba69b000 SMBIOS 3.0=0xba698000 ACPI=0xbb5fe000 ACPI 2.0=0xbb5fe014 MEMATTR=0xb4a1f298
 Feb 19 10:08:21 mdbauman kernel: Secure boot enabled and kernel locked down
 Feb 19 10:08:21 mdbauman kernel: Bluetooth: hci0: Secure boot is enabled
 Feb 19 10:18:41 mdbauman fwupd[2852]: disabling plugin because: failed to coldplug uefi: UEFI firmware updating not supported
InstallationDate: Installed on 2017-09-09 (162 days ago)
InstallationMedia: Ubuntu 17.10 "Artful Aardvark" - Alpha amd64 (20170926)
SecureBoot: 6 0 0 0 1
SourcePackage: shim
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Mathieu Trudel-Lapierre (cyphermox)]

I really don't think you're affected by the same issue as the other bug.

I see there's a BootOrder set (at least, once booted), and there's an ubuntu entry that looks correct, but that does depend on what you're supposed to be booting to. The screenshot you include says the system should be booting from NVMe, but the BootEntry for Ubuntu is not quite looking the same.

What process did you take to boot to Ubuntu to file the bug?

Usually, for NVMe you should see a BootEntry that includes "nvme("; doesn't seem to be the case here. Maybe this particular firmware only likes to use UUIDs as we see there?

[Moritz Baumann (mb-o)]

When I first encountered the problem, I first tried to let boot-repair (run from a live system) repair my Grub installation, since I'm not too familiar with the boot process on UEFI. It failed multiple times, even when I disabled Secure Boot and when I enabled the CSM and tried to install grub-pc.

In the end, the only solution that worked was to chroot into the installed system, purge all grub and shim packages and install the original versions. This is the current state of my installation:

> apt-mark showhold
grub-common
grub-efi-amd64
grub-efi-amd64-bin
grub-efi-amd64-signed
grub2-common
shim
shim-signed

> apt list --upgradable
Listing... Done
grub-common/artful-updates 2.02~beta3-4ubuntu7.1 amd64 [upgradable from: 2.02~beta3-4ubuntu7]
grub-efi-amd64/artful-updates 2.02~beta3-4ubuntu7.1 amd64 [upgradable from: 2.02~beta3-4ubuntu7]
grub-efi-amd64-bin/artful-updates 2.02~beta3-4ubuntu7.1 amd64 [upgradable from: 2.02~beta3-4ubuntu7]
grub-efi-amd64-signed/artful-updates 1.85.1+2.02~beta3-4ubuntu7.1 amd64 [upgradable from: 1.85+2.02~beta3-4ubuntu7]
grub2-common/artful-updates 2.02~beta3-4ubuntu7.1 amd64 [upgradable from: 2.02~beta3-4ubuntu7]
shim/artful-updates 13-0ubuntu2 amd64 [upgradable from: 0.9+1474479173.6c180c6-1ubuntu1]
shim-signed/artful-updates 1.33.1~17.10.1+13-0ubuntu2 amd64 [upgradable from: 1.32+0.9+1474479173.6c180c6-1ubuntu1]

I'm sorry that I cannot provide more details here due to my lack of knowledge regarding the UEFI boot process. If you tell me what to look for, I can try to provide more information. And now that I know how to restore my system, I'd be happy to try and collect some information with the buggy versions installed.

[Mathieu Trudel-Lapierre (cyphermox)]

The packages themselves are not buggy. This appears to be an issue caused by firmware doing exactly what it is told and designed to do.

I didn't notice that earlier, but there's a "Boot Order Lock" option in Lenovo's firmware, and it is set to Enabled in the screenshots you provided.

Automated boot fallback can only work if the BootOrder can be changed, as that is what it used to let the system boot directly to disk, then rebuild the 'ubuntu' entry, and reboot to get into it.

If you disable 'Boot Order Lock' and remove the hold on the packages, I expect the system to successfully update, and boot to Ubuntu successfully. At the very least, it will definitely help ensuring fallback works without multiple reboot.

I'm still not sure if there isn't also a latent issue caused by NVMe, but I think this will get us one step in the right direction :)

[Moritz Baumann (mb-o)]

Thank you Mathieu, your suspicion was correct: After disabling that setting, the message on boot is gone, the latest versions of the packages work and I now see an "ubuntu" entry next to the NVMe entry in the UEFI boot device menu.

This resolves my immediate issue. I guess the actual problem is that I did not see an error message telling me that the "ubuntu" entry could not be written and that I'd have to modify my UEFI settings (or that this fallback was activated in the first place?). Do you need any more information from my side?

[Mathieu Trudel-Lapierre (cyphermox)]

Not really; the error not showing up is because the firmware doesn't report an error, but even if it did we can't necessarily tell the user anything from it anyway (it might not be breaking because of Boot Order Lock).

Fallback is a "new" feature to fix other systems that don't remember how to boot across reboots...

Closing as "Invalid": this is effectively a support request, where firmware configuration caused fallback to fail.