firmware upgrades remove the ubuntu boot loader from EFI boot vars

[Marius Gedminas]

snap-store offered me a system firmware upgrade for my Lenovo ThinkPad X390. I installed it with fwmgr upgrade [*], which prompted me to reboot, which I did, and I saw the firmware upgrade go through, and then it rebooted again -- into Windows.

The UEFI boot menu I get on F12 offers me Windows, the NVME drive (which boots Windows), PXE boot, and a Linux Firmware Installer. If I try that one, I get an error about 0 firmware updates being available and the system hangs until I hold the power button.

I've booted from a rescue USB, checked efibootmgr -v, saw no Ubuntu entry there, re-ran grub-install /dev/nvme0n1, and now the system boots normally.

[*] I tried to install the update from snap-store directly, but my battery was at 7% so that failed with a notification stating the reason. I waited until the battery was charged up to 15%, then tried again, but this time the update failed without giving me a reason. Then I did the update from a terminal with fwupdmgr.

ProblemType: Bug
DistroRelease: Ubuntu 20.04
Package: fwupd 1.3.11-1~focal1
ProcVersionSignature: Ubuntu 5.4.0-51.56-generic 5.4.65
Uname: Linux 5.4.0-51-generic x86_64
ApportVersion: 2.20.11-0ubuntu27.9
Architecture: amd64
CasperMD5CheckResult: skip
CurrentDesktop: ubuntu:GNOME
Date: Fri Oct 16 13:13:37 2020
EcryptfsInUse: Yes
InstallationDate: Installed on 2019-06-12 (491 days ago)
InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
SourcePackage: fwupd
UpgradeStatus: Upgraded to focal on 2020-04-24 (174 days ago)
mtime.conffile..etc.fwupd.remotes.d.lvfs-testing.conf: 2020-10-08T10:00:47.263185
mtime.conffile..etc.fwupd.remotes.d.lvfs.conf: 2020-10-08T10:00:47.311185

[Mario Limonciello (superm1)]

The behavior of what boot entries are used or changed after a firmware upgrade is entirely a firmware behavior.

@Mark can you please have your team look into this.

[Mark Pearson (markrhpearson)]

Definitely. Thanks Mario

Mark

[Mark Pearson (markrhpearson)]

Note - tracked internally by LO-788 (for my reference)

[Marius Gedminas (mgedmin)]

A strange thing happened today: snap-software told me there are important system upgrades available, and one of those was the same firmware upgrade for the x390 (1.67 -> 1.70) that I was sure I'd already installed yesterday.

fwupdmgr get-updates and /sys/class/dmi/id/bios_version confirmed that no, my firmware was still at version 1.67, despite me seeing the upgrade process with my own eyes yesterday.

So I installed the update again, took a snapshot of efibootmgr -v for my notes, saw that it had all the right entries (BootNext: 0002 aka Linux-Firmware-Updater, BootCurrent: 0001 aka ubuntu), rebooted, saw the firmware upgrade process again (two capsules, nice large translated installing text in the middle of the screen with a |/- spinner above it overlapping the huge LENOVO logo, then a red progress bar, then another reboot, then a black screen with a "self-healing bios backup" and an increasing percentage text, then another reboot.

I went into the boot order menu (F12) and saw that ubuntu was still there. I booted and /sys/class/dmi/id/bios_version is showing 1.70 this time.

I have no explanation, only theories. Maybe I did something yesterday (boot into the Linux Firmware Upgrade target with no upgrade queued?) that made the laptop think the firmware got corrupted and it restored the older version from a backup?

[Mario Limonciello (superm1)]

Perhaps a relevant detail: https://github.com/fwupd/fwupd/pull/2757

[ErgoWang (ergo-wang001)]

Hi Marius.I want to confirm some details：
   1.Could you provide me specific firmware (e.g. BIOS、ME……) that snap-store offered.
   2.Do you still remember the version of the firmware before and after upgrade.
   3. As description, you fix it through one instruction from “rescue USB”. Could you provide this tool, it helps to fix issue.

[Marius Gedminas (mgedmin)]

1. My notes say "LENOVO System firmware 0.1.67 -> 0.1.70, also described as Lenovo ThinkPad T490s/ThinkPad X390 (W-BIOS) System Firmware 1.70".

I have recorded in my notes that after I attempted the update from snap-store (at 7% battery), I ran fwupdmgr get-upgrades in a terminal and saw

  │ Update State: success
  │ Update Error: failed to update_prepare using upower: Cannot install update when battery is not at least 10% unless forced

and then I checked the battery level again (the laptop was plugged in at the time), saw it had risen to 12%, so I ran

  fwupdmgr upgrade

and restarted when prompted. (And then saw the firmware upgrade process, and rebooted again, and got Windows, with no Ubuntu in the UEFI boot menu, after which I proceeded to disable secure boot and boot from my USB key).

2. `fwupdmgr get-history` today shows

├─System Firmware:
│ │ Device ID: ca8e899e46f1da1776d16410bc92f2cf0d963433
│ │ Previous version: 0.1.67
│ │ Update State: success
│ │ Update Error: failed to update to 117474352: battery level is too low
│ │ Last modified: 2020-10-17 06:17
│ │ GUID: 6300ba52-d9ea-4ca0-88f9-f89ef73cc07b
│ │ Device Flags: • Internal device
│ │ • Updatable
│ │ • Requires AC power
│ │ • Supported on remote server
│ │ • Needs a reboot after installation
│ │ • Reported to remote server
│ │ • Cryptographic hash verification is available
│ │ • Device is usable for the duration of the update
│ │
│ └─ThinkPad T490s/X390 (W-BIOS for Machine types: 20NX, 20NY, 20Q0, 20Q1) System Update:
│ New version: 0.1.70
│ Remote ID: lvfs
│ Summary: Lenovo ThinkPad T490s/ThinkPad X390/ThinkPad System Firmware
│ License: Proprietary
│ Size: 25,1 MB
│ Created: 2020-08-31
│ Urgency: High
│ Vendor: Lenovo Ltd.
│ Description: Lenovo ThinkPad T490s/ThinkPad X390 (W-BIOS) System Firmware 1.70
│
│ The computer will be restarted automatically after updating BIOS completely. Do NOT turn off your computer or remove the AC adaptor while update is in progress.
│
│ This stable release fixes the following issues:
│
│ • Fixed an issue where BIOS might generate 1802 error with L850 WWAN.
│ • Fixed an issue where system may occur post hang issue with TBT dock.
│ • Fixed an issue where MEFW LVFS file can't be updated.

3. That was my USB drive with a custom grub.cfg that lets me pick one of several Ubuntu ISO images. https://mg.pov.lt/blog/booting-iso-from-usb-2020.html has more information, but really, all I did was disable secure boot, select the USB drive in the UEFI boot menu, get me a GRUB menu, cancel it, get the GRUB command line and do a bunch of ls (hd1,gpt1)<tab><tab> to figure out where everything is, then chain-boot with

    grub> configfile (hd1,gpt5)/boot/grub/grub.cfg

to get the actual grub config from the root partition o...

[Marius Gedminas (mgedmin)]

If it helps, I have in my notes the full output of `efibootmgr -v` from Oct 16 with the 'ubuntu' entry missing, and then the full output after I ran grub-install to recover.

I do not have a full output of `efibootmgr -v` from earlier, but I do have just the `ubuntu` entry from February, after a little misadventure with creating UEFI-bootable USB drives messed up my EFI boot vars and I successfully recovered by running `grub-install`.

[ErgoWang (ergo-wang001)]

Hi Marius.
   I spent some time following the steps and testing them, but the phenomenon you describe didn't happen. After charging battery to 12% and updating by "fwupdmgr upgrade", computer restart -> show "Check power failure,please insert the power source or charge the power" -> power on normally.

Except for the information you provided, I'd like to confirm other details:
   Have you ever entered BIOS (or Setup) and pressed "F9". It will excute "Load default"(or "Setup default").Althrough it seem insignificant, that may cause the issue you mentioned.

   By the way, althrough you charge battery to 12%, the bios upgrades still failed due to low power. When the battery is above 50% ,bios will update successfully.

[Marius Gedminas (mgedmin)]

> After charging battery to 12% and updating by "fwupdmgr upgrade", computer restart -> show "Check power failure,please insert the power source or charge the power" -> power on normally.

The error message give the impression that you had the power cable disconnected and battery at 12% while trying to update the firmware? When I tried this, the power cable was plugged in.

(Possibly the error message is just misleading. I regret not paying close attention when I did that the first time, so I didn't notice any error messages that might have been displayed, as long as they didn't require a keypress to dismiss.)

> Have you ever entered BIOS (or Setup) and pressed "F9". It will excute "Load default"(or "Setup default").

No, I haven't.

> By the way, althrough you charge battery to 12%, the bios upgrades still failed due to low power. When the battery is above 50% ,bios will update successfully.

Oh, that is interesting information and it explains one mystery at least!

[Marius Gedminas (mgedmin)]

This happened again, while trying to upgrade the Lenovo ThinkPad X390 system firmware from version 1.70 to 1.71. I missed the initial details again, unfortunately -- didn't see what the computer was doing while rebooting, came back to see it booted into Windows with the 'ubuntu' boot menu entry missing from EFI vars, and the firmware update not applied.

[Marius Gedminas (mgedmin)]

I'm now reliably in this situation:

efibootmgr shows

    | BootCurrent: 0003
    | Timeout: 0 seconds
    | BootOrder: 0003,001A,0000,0018,0019,001B,001C,001D,001E,001F,0020,0021,0002,0027
    | Boot0000* Windows Boot Manager
    | Boot0002* Linux-Firmware-Updater
    | Boot0003* ubuntu
    ...

I run fwupdmgr upgrade, then check efibootmgr before the reboot and see

    | BootNext: 0001
    | BootCurrent: 0003
    | Timeout: 0 seconds
    | BootOrder: 0003,001A,0000,0018,0019,001B,001C,001D,001E,001F,0020,0021,0002,0027,0001
    | Boot0000* Windows Boot Manager
    | Boot0001* Linux-Firmware-Updater
    | Boot0002* Linux-Firmware-Updater
    | Boot0003* ubuntu

then I reboot and see the grub menu instead of the firmware upgrade, and when I boot I see

    | BootCurrent: 0003
    | Timeout: 0 seconds
    | BootOrder: 0003,001A,0000,0018,0019,001B,001C,001D,001E,001F,0020,0021,0002,0027
    | Boot0000* Windows Boot Manager
    | Boot0002* Linux-Firmware-Updater
    | Boot0003* ubuntu

If I try to manually boot the Linux Firmware Installer boot entry, I get a bunch of errors:
https://twitter.com/mgedmin/status/1385605275948830720/photo/1

(The reason why I had two Linux-Firmware-Updater entries is bug 1925813. I got ubuntu on Boot0003 after I manually ran grub-install while Boot0001 and Boot0002 were both Linux-Firmware-Updater.)

[Marius Gedminas (mgedmin)]

I did a fundamental mistake and changed two things at a time:

1. deleted the duplicate Linux-Firmware-Updater boot entry with efibootmgr -B -b 0002
2. decided to apply the two firmware upgrades separately

I ran fwupdmgr upgrade twice, selected the Intel ME upgrade first, rejected the System Firmware upgrade, rebooted, saw the upgrade being applied, after which the laptop booted into Ubuntu.

I ran fwupdmgr upgrade again, let it upgrade the system firmware, rebooted, and it worked fine.

I wonder if my mistake was trying to upgrade both?

I wonder if it was the system firmware deleting Boot0001 both initially (when that was pointing to grub) and in comment #13 (when that was pointing to Linux-Firmware-Updater) when it saw a bunch of capsules it didn't like?

efibootmgr currently looks like this:

    | BootCurrent: 0003
    | Timeout: 0 seconds
    | BootOrder: 0003,001A,0000,0018,0019,001B,001C,001D,001E,001F,0020,0021,0027,0001
    | Boot0000* Windows Boot Manager
    | Boot0001* Linux-Firmware-Updater
    | Boot0003* ubuntu

and it looked mostly the same before and after each reboot, with just a BootNext: 0001 showing up after each fwupmgr upgrade call. Nothing is removing Boot0001 when the firmware upgrades are successful.

[Mario Limonciello (superm1)]

Closing this task in Ubuntu as this is a FW bug - there is no work for fwupd as far as I can tell. If that ends up being wrong we can re-open. If necessary please discuss further in https://github.com/fwupd/firmware-lenovo or Lenovo forums.