shim 15+1552672080.a4a1fbe-0ubuntu1 fails to load fwupd

[Julian Andres Klode]

(this is a regression of shim in groovy, fixed by new upstream release, it can serve as regression test case for 15+1552672080.a4a1fbe-0ubuntu2 SRU, the SRU itself is tracked in bug 1862171)

The latest shim upload does not seem able to load fwupd. Selecting fwupd in BIOS boot menu seems to go directly to grub.

Probably not a signing issue of fwupd, as we don't get a security violation error. Need to investigate more.

[Eugene Crosser (crosser)]

I think that it happens to me, and it's regression going from eoan to focal. Firmware updates are downloaded and installed, and even if I set BootNext by hand, it is bypassed.

Linux-Firmware-Updater looks suspicious to me:

# efibootmgr -v|grep Linux-Firmware-Updater
Boot0002* Linux-Firmware-Updater HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)\.f.w.u.p.d.x.6.4...e.f.i...

Is ".f.w.u.p.d.x.6.4...e.f.i" supposed to look like that?

[Julian Andres Klode (juliank)]

Yes

[Eugene Crosser (crosser)]

To clarify my comment #1: update files are installed in /boot/efi/EFI/ubuntu/fw, but on reboot, there is no sign of fwupdx64.efi ever running, boot process goes straight to grub.

[Julian Andres Klode (juliank)]

Right, tracking this in https://github.com/rhboot/shim/issues/194

[Steve Langasek (vorlon)]

Mario, you've set a beta milestone for this but if this requires changes to the shim source, that is not achievable without us rolling back to an earlier version of shim-signed. We are not going to have this fixed and re-signed by Microsoft in time for beta.

[Mario Limonciello (superm1)]

@Steve

Yes; from what I can gather this will definitely require source modifications to shim.

The concern I have is that beta is the milestone that many more people start to download and actually start testing Ubuntu images. With how widely OEMs support UEFI firmware updates now, I expect a larger influx of bugs to be reported around failing firmware updates as people load the beta images on their machines.

If it's not possible to fix this particular issue by the beta milestone but the shim changes are preferable to keep in, I wonder if it would make sense to make some modifications to fwupd. Some alternative idea proposals:

1) If secure boot is not turned on, don't build the "Linux Firmware Updater" entry to use shim "at all". Instead BDS would load fwupdx64.efi directly. This would prevent hitting this particular bug if secure boot was turned off.
2) If secure boot is turned on, detect the version of shim on the system at runtime from fwupd and add a blacklist of this particular shim version so that updates are not offered.

Those would both require some source modifications to fwupd, but I think they're achievable workarounds by beta milestone.

[Mario Limonciello (superm1)]

As confirmed on the upstream bug:

Commit https://github.com/rhboot/shim/commit/e563bc3dcd17d91861d3b363ed19d30228f409e1 caused the issue, and it was already fixed by upstream commit https://github.com/rhboot/shim/commit/1870bae796022f8bbf60465352eac329ff1d6ffd

[Alex Murray (alexmurray)]

I have rebuilt shim from focal with the suggested patch from upstream and can confirm that with the resulting shimx64.efi (and Secure Boot disabled) shim would successfully launch fwupd as expected and pending firmware updates were successfully installed. See attached for the debdiff used in this case.

[Julian Andres Klode (juliank)]

We already have an ubuntu2 in the shim PPA. It's blocked by

https://github.com/CanonicalLtd/shim-review/pull/1

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "shim_15+1552672080.a4a1fbe-0ubuntu2.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

[Julian Andres Klode (juliank)]

Removing the sponsors team, as there's nothing to sponsor here.

[Eugene Crosser (crosser)]

If there is an updated package available, I am willing to test it on my system..

[Julian Andres Klode (juliank)]

Let's be clear: I don't want you to test it - You need a machine w/ secure boot disabled, and need to replace shim-signed with shim and manually replace the shim binary to the ESP. You'll end up with an unsupported system.

That said, it is in its usual location, the uefi development ppa:

https://launchpad.net/~ubuntu-uefi-team/+archive/ubuntu/ppa

Next steps are:

- Merge the merge proposal
- Get shim-review board approval
- Get shim signed by microsoft

If we're lucky, we get this done before the release, that said, the last time, this took 4 months (October to February).

[Eugene Crosser (crosser)]

@Julian, of course I understand the implications. Thank you for the pointer.
(Is it an option to revert to previous signed shim for the release? It did work.)

[Mario Limonciello (superm1)]

@juliank,

If it's unlikely that this is done before the release can we revert back to the older shim for release and bump up to this new one after it comes through?

[Jonathan Kamens (jik)]

I think this has been fixed in focal? Within the last few days (I believe) my laptop successfully installed the firmware updates that had been failing to install because of this issue.

Which is weird because the bug says it's in shim, but my dpkg log says the last time the shim package was updated was in February? Maybe it wasn't actually a shim bug?

[Julian Andres Klode (juliank)]

We reverted the shim upload in focal. A fixed one will be released after focal release.

[Eugene Crosser (crosser)]

I see 1.40+15+1533136590.3beb971-0ubuntu1 in the main focal repo, and 15+1552672080.a4a1fbe-0ubuntu1 in -proposed. Neither of them load fwupdx64.efi for me?

[Christopher Townsend (townsend)]

I have shim 15+1552672080.a4a1fbe-0ubuntu1 and shim-signed 1.41+15+1552672080.a4a1fbe-0ubuntu1 installed on my 7th gen Lenovo Carbon X1 and it still fails to boot the firmware updater properly.

[Julian Andres Klode (juliank)]

Yes that's what the bug report is about - those are the broken versions. We reverted to older ones. You don't get downgraded to them if you caught the broken one, it's not technically possible.

[Julian Andres Klode (juliank)]

Eugene I'm sure you misconfigured something. You need to downgrade shim-signed (not only shim).

[Jonathan Kamens (jik)]

If "you don't get downgraded to [the working versions] if you caught the broken one," then can someone explain to me how my laptop suddenly successfully upgraded two different firmware files after it failed to do so for weeks presumably because of this bug?

[Eugene Crosser (crosser)]

Could it be that newer "bad" versions continue to linger on ubuntu mirrors?

Julian, can you post here, which version is known to work and is going to the official distro?

[Julian Andres Klode (juliank)]

The bad versions have been demoted to proposed, and were removed recently. You need a shim-signed 1.40.something installed, 1.41 was broken.

[Julian Andres Klode (juliank)]

jik, I can't tell you what's going on on your device.

[Christopher Townsend (townsend)]

Oh, you have to manually downgrade the packages to what's in main now? I guess that should be documented somewhere for those of us who are affected by this since it's not obvious (at least to me:)) that's what one needs to do.

[Christopher Townsend (townsend)]

Meant to sat "thanks" in my last post too:)

Anyways, here is the command to downgrade the packages if you are affected by this:

$ sudo apt install shim=15+1533136590.3beb971-0ubuntu1 shim-signed=1.40+15+1533136590.3beb971-0ubuntu1

Thanks again!

[Eugene Crosser (crosser)]

crosser@journex:~$ apt policy shim-signed
shim-signed:
  Installed: 1.40+15+1533136590.3beb971-0ubuntu1
  Candidate: 1.40+15+1533136590.3beb971-0ubuntu1
  Version table:
 *** 1.40+15+1533136590.3beb971-0ubuntu1 500
        500 http://ftp.tu-ilmenau.de/mirror/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

sha256sum:
de885a99face77a23bf217a54da716537a3f9d8112384054fb46785c4a2ee388 /usr/lib/shim/shimx64.efi.signed
de885a99face77a23bf217a54da716537a3f9d8112384054fb46785c4a2ee388 /boot/efi/EFI/ubuntu/shimx64.efi

Does not work for me :(
After re-running firmware updater, it reboots and goes directly to grub

[Mario Limonciello (superm1)]

@crosser:

If you're matching the version of shim and shim-signed in the archive now and hitting this problem you have a different issue. It's notable that your manufacturer has a firmware option for boot order locking that is commonly causing similar behaviors. If you have enabled it, you may want to explore disabling it to try to help this issue.

[Eugene Crosser (crosser)]

I did not change any bios settings since upgrade from eoan, and firmware update worked in eoan.

But I found something interesting.

In "normal life" my efi configuration looks like this:

BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0019,001A,001B,001C,001D,001E,001F,0020,0021,0022,0023,0024,0002
Boot0001* ubuntu HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)
Boot0002* Linux-Firmware-Updater HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)\.f.w.u.p.d.x.6.4...e.f.i...
Boot0010 Setup FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011 Boot Menu FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012 Diagnostic Splash Screen FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
...

Note that there is an entry for fwupdx64 number 0002.
After running fwupdmgr, configuration looks like this:

BootNext: 0000
BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0019,001A,001B,001C,001D,001E,001F,0020,0021,0022,0023,0024,0002,0000
Boot0000* Linux-Firmware-Updater HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)\.f.w.u.p.d.x.6.4...e.f.i...
Boot0001* ubuntu HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)
Boot0002* Linux-Firmware-Updater HD(1,GPT,6ccce482-e2c2-48ca-991e-608bee5d38af,0x800,0x100000)/File(\EFI\ubuntu\shimx64.efi)\.f.w.u.p.d.x.6.4...e.f.i...
Boot0010 Setup FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0011 Boot Menu FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0012 Diagnostic Splash Screen FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
...

Note that fwupdmgr added a new entry for Linux-Firmware-Updater number 0000, and set it for BootNext. So now there are two entries for Linux-Firmware-Updater, that looks the same. After reboot, updater _did not_ run, and efi configuration returned to "normal" state.

After that, I set BootNext to 0002 by hand, and rebooted. And now, firmware updater _worked_!

Another piece of information: replacing shim-signed actually _did_ make a difference. I tried setting BootNext to 0002 with the previous focal shim, and it did _not_ run updater. Now it does.

So maybe there is a second problem, with fwgupdmgr that sets a second entry? Or there is another problem in the shim, that it does not honour the second entry or entry number 0000 in efi configuration?...

Tell me if you need more information from me.

[Mario Limonciello (superm1)]

OK thanks for confirming it's not cause by boot order lock.

I think that your scenario is a separate issue from the shim one entirely. I would recommend bringing it to discussion upstream to figure out why the entry didn't get re-used. It could be a firmware issue too.

[Eugene Crosser (crosser)]

Before raising the issue with fwupdmgr, why did not shim load the updater from the entry 0000 as instructed by BootNext? Configuration looks strange, but legit to the eye. Does shim or firmware check for duplicate entries? Does shim or firmware not allow to use entry 0?

Right now, I am unsure what is that "upstream" where I am supposed to bring the discussion. As far as I can see the symptoms, shim is refusing to load fwupdx64.efi for unclear reason. And once again, firmware update _did_ work in eoan!

[Mario Limonciello (superm1)]

From your circumstances I don't believe there is any evidence to support that shim attempted to load fwupd, nor evidence that boot entry was even used.

The only way to prove this would be to look at what "BootCurrent" was set to after a "failed" attempt. If it's set to Boot0001 then your firmware did not respect BootNext or at least felt that the Boot0000 was not a valid entry.

If it's set to Boot0002 or Boot0000 then that means that shim loaded, but didn't load the correct binary.

The reason that I think upstream fwupd should look at this is because the behavior here did change slightly in fwupd 1.3.8. Previously the entries were deleted after flashing firmware, but now in 1.3.9 they're supposed to be re-used.
You can see this commit that removed the behavior of deleted entries:
https://github.com/fwupd/fwupd/commit/60373e03fd0668d379ba0d7b90a6f02ba7d87478#diff-2f5c39c775e4cac7ebbeabed4cefca6a

So I have a hypothesis that your Boot0002 entry was not selected sounds by a combination of both a firmware bug (handling duplicate entries) and a fwupd bug (not viewing it as a valid entry since Boot0000 was empty).

[Eugene Crosser (crosser)]

Thanks Mario. The theory makes sense.
I _think_ that after boot with BootNext=0000, entry 0000 disappeared and BootCurrent was 0001 (but I cannot be 100% sure as I did not save the config at the time).

My firmware _did_ respect BootNext when it was set to 0002, and it was the only updater entry.

I will try to bring this to attention of fwupd.

[Mario Limonciello (superm1)]

If the entry disappeared on it's own, that sounds like it's probably the firmware's doing in somnething like "de-duping" entries. And it would explain this behavior better. Something along the lines of:
1) Reboot system
2) Firmware runs de-dupe algorithm
3) Firmware tries to run BootNext (fails)
4) Firmware runs BootOrder as a fallback (which was 0001).

So if fwupd had chosen to re-use 0002 this wouldn't have exposed your firmware bug.

[Andreas Hasenack (ahasenack)]

I don't have boot order lock.

shim is 15+1552672080.a4a1fbe-0ubuntu1, shim-signed is 1.41+15+1552672080.a4a1fbe-0ubuntu1

Applying firmware updates isn't working. I reboot, nothing different happens, I'm back in the desktop, and get another prompt to apply the same firmware update.

[Christopher Townsend (townsend)]

Hey @ahasenack,

You have the "broken" versions installed. You need to do the steps in https://bugs.launchpad.net/ubuntu/+source/shim/+bug/1864223/comments/27.

[Julian Andres Klode (juliank)]

People who had focal installed and uptodate prior to the beta (roughly) won't get this working again until we can SRU a fixed shim binary or they manually downgrade to the old versions in the release pocket.

[Julian Andres Klode (juliank)]

This is a technical limitation, because we can't re-upload shim with a higher version to downgrade people automatically, as we would not have a signature for the binaries that upload would built.

And shim-signed has a versioned dependency on shim, as it needs to, so we could not just bump the version there - apt would hold it back.

[Andreas Hasenack (ahasenack)]

Ah, indeed, the "broken" one isn't even in the archive anymore, so there is nothing I can upgrade to:

$ apt-cache policy shim shim-signed
shim:
  Installed: 15+1552672080.a4a1fbe-0ubuntu1
  Candidate: 15+1552672080.a4a1fbe-0ubuntu1
  Version table:
 *** 15+1552672080.a4a1fbe-0ubuntu1 100
        100 /var/lib/dpkg/status
     15+1533136590.3beb971-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
shim-signed:
  Installed: 1.41+15+1552672080.a4a1fbe-0ubuntu1
  Candidate: 1.41+15+1552672080.a4a1fbe-0ubuntu1
  Version table:
 *** 1.41+15+1552672080.a4a1fbe-0ubuntu1 100
        100 /var/lib/dpkg/status
     1.40.3+15+1533136590.3beb971-0ubuntu1 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

[Christopher Townsend (townsend)]

Right, you have to force it to install the older version as I mentioned in that post. Not sure if you did that or not based on your reply:)

[Seth Arnold (seth-arnold)]

I don't think that this broken package was announced widely enough -- the fallout from it cost my team perhaps an hour today. I don't know where this was announced, but I think probably more places needed to hear about it at the time.

Whoever works on shim, *please* announce future cases like this everywhere, to the point that we're all sick of hearing about it. :)

Thanks

[Christopher Townsend (townsend)]

@seth-arnold +1

[Launchpad Janitor (janitor)]

This bug was fixed in the package shim - 15+1552672080.a4a1fbe-0ubuntu2

---------------
shim (15+1552672080.a4a1fbe-0ubuntu2) focal; urgency=medium

  * d/patches/fix-path-checks.patch: Cherry-pick upstream fix for regression
    in loading fwupd, or anything else specified as an argument (LP: #1864223)

 -- Julian Andres Klode <email address hidden> Fri, 20 Mar 2020 16:19:14 +0100

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted shim into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/shim/15+1552672080.a4a1fbe-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted shim into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/shim/15+1552672080.a4a1fbe-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Łukasz Zemczak (sil2100)]

Hello Julian, or anyone else affected,

Accepted shim into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/shim/15+1552672080.a4a1fbe-0ubuntu2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

Marking this as verification done. This was a regression in focal(devel) regarding path lookup that was reverted prior to release, we have validated that this works in groovy, and the binary is the same across all releases, so this does not need per-series re-verification.

[Launchpad Janitor (janitor)]

This bug was fixed in the package shim - 15+1552672080.a4a1fbe-0ubuntu2

---------------
shim (15+1552672080.a4a1fbe-0ubuntu2) focal; urgency=medium

  * d/patches/fix-path-checks.patch: Cherry-pick upstream fix for regression
    in loading fwupd, or anything else specified as an argument (LP: #1864223)

 -- Julian Andres Klode <email address hidden> Fri, 20 Mar 2020 16:19:14 +0100

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for shim has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package shim - 15+1552672080.a4a1fbe-0ubuntu2

---------------
shim (15+1552672080.a4a1fbe-0ubuntu2) focal; urgency=medium

  * d/patches/fix-path-checks.patch: Cherry-pick upstream fix for regression
    in loading fwupd, or anything else specified as an argument (LP: #1864223)

 -- Julian Andres Klode <email address hidden> Fri, 20 Mar 2020 16:19:14 +0100