arm64 Secure Boot fails w/ "error: cannot load image."

[dann frazier]

[Impact]
Kernels are not loaded on arm64 since grub2 2.04 rebase

[Test case]

1. Boot qemu

qemu-system-aarch64 -m 1024 -cpu cortex-a53 -M virt -pflash ~/AAVMF_CODE.fd -pflash ~/AAVMF_VARS.fd -drive if=none,file=/home/jak/Downloads/focal-server-cloudimg-arm64.img,id=hd0,format=qcow2 -device virtio-blk-device,drive=hd0 -net user,hostfwd=tcp::10022-:22 -net nic -smp 8 -device ramfb -device nec-usb-xhci -device usb-tablet -device usb-kbd

2. Run sb-setup enroll microsoft from https://git.launchpad.net/qa-regression-testing/tree/notes_testing/secure-boot after sed s#x86_64#arm64#g s#x64#aa64#g to enroll the keys and put machine into secure boot mode

3. Verify that new grub works

4. Parallel tests (take snapshot after 3)
4a) Reinstall old grub and check that it fails
4b) Remove kernel signature and check that kernel loading fails

[Regression potential]
The changes are limited to grub-core/loader/arm64/linux.c, hence we can only break booting on non-secureboot arm64 (where it worked before) or introduce a security regression on secure boot systems (in case there's a bug). I've checked the code against fedora-34 patches, and did not see anything that looks like trouble, though, also we had it in bionic.

[Original bug report]
I tested out the new shim-signed (1.41+15+1552672080.a4a1fbe-0ubuntu1) on arm64 today. Unfortunately, I was unable to boot a kernel. I tried manually running commands in the GRUB shell to try and get more info, and here's the error I get:

grub> insmod gzio
grub> linux (hd0,gpt1)/boot/vmlinuz-5.4.0-13-generic
grub> boot
error: cannot load image.

This is better then it was previously - shim used to crash before starting GRUB (bug 1811901 and bug 1811722). But obviously there are still issues somewhere. Prior to this shim binary being signed, I believe I had tested the unsigned binary in a VM using a custom signing certificate. I think I still have that VM around, so I maybe able to use it for comparison.

= My setup =
I tried to make this test simulate a real setup as much as possible. Here's roughly what I did:

Installed an arm64 server w/ bionic
# need a new QEMU for EnrollDefaultKeys.efi
sudo apt-add-repository cloud-archive:train
sudo apt update
sudo apt install uvtool
sudo gpasswd -a ubuntu libvirt
# log out/back in
# no focal images yet
uvt-simplestreams-libvirt -v sync release=eoan
uvt-kvm create focal arch=arm64 release=eoan
uvt-kvm wait focal
uvt-kvm ssh focal
guest> sudo sed -i 's/eoan/focal/' /etc/apt/sources.list
guest> # Also enabled focal-proposed to get latest shim-signed
guest> sudo apt update
guest> sudo apt dist-upgrade
guest> sudo apt install shim-signed
guest> sudo grub-install
# On an x86 host, I built the latest edk2 package and copied out the AARCH64 build of
# EnrollDefaultKeys.efi. I scp'd this over to the focal guest, and put it in the EFI
# system partition
guest> sudo poweroff
virsh edit focal
# Add the following to inject the Pk/KEK keys:
# <qemu:commandline>
# <qemu:arg value='-smbios'/>
# <qemu:arg value='type=11,value=4e32566d-8e9e-4f52-81d3-5bb9715f9727:MIIDNjCCAh4CCQCUy69JzVan2DANBgkqhkiG9w0BAQsFADBdMS0wKwYDVQQDDCRVYnVudHUgT1ZNRiBTZWN1cmUgQm9vdCAoUEsvS0VLIGtleSkxLDAqBgkqhkiG9w0BCQEWHXVidW50dS1kZXZlbEBsaXN0cy51YnVudHUuY29tMB4XDTE4MDYyMDIxNDg0NloXDTI4MDYxNzIxNDg0NlowXTEtMCsGA1UEAwwkVWJ1bnR1IE9WTUYgU2VjdXJlIEJvb3QgKFBLL0tFSyBrZXkpMSwwKgYJKoZIhvcNAQkBFh11YnVudHUtZGV2ZWxAbGlzdHMudWJ1bnR1LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMuwK+l3nl5x6ebrHYVShs/7jPAKeTTMu4MQlTbNoOZvVQhOcedjkBNaPPdd63TBxYFAnJhUBLl9hW/GB5Fn9itT0yh5G64XCBafy3rJLF8L99VDUYEuvB+a3boYATCToVnODb8h0ImORBF8sgKZm65CJlgQ93YGZbjLePnuawhU2EVH2HFyLZEWjd3JPxstlzGj+JiwvETdFX/fHbnrW+fLCLEnLLZ/YPo6We0mtVTEqHWm6G5WUIbpzPzOOGpiCKHdI+VFsX7w1TBdMhCqnxcpLn7NRXEEgw+OQ5gnOLR9kTKI+MRkux9pDGZ5v9VMcPZi2iZTHRd9briIGOL/fo0CAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAGLAtUs7fnf5oKU7E7+woUrHP03WXAwhTNI9eTs7YLPgwC2qGAGkzdUZUbzc4zS4SaItITlYYeWfZ9PvPhPGyIZOeuBMoUeBknsC2daRVX11aAcgOnQhxMD0WjSRG5nQ5rXRZ/NwYvctJR81l41kDToNqjBIjJ3FThzz8hHyMv/DCh3ch/X2Hj7ib+1IPfoHFk+mD/6e+y46wHWS5u0Bol9w4VBMwa3FYniFgKrAmnoiuo2br5fBbgH/7326lJ7Qb/H4mBLKz/c3iw4PF+KQxspc04tJdvQ+pDEtTUiXVE0zcBip2EJgPVK0szO5H6gtXbfyoTqDr1DKaD4x9JD3yKQ=='/>
# </qemu:commandline>
#
virsh start focal; virsh console focal
# Interrupt focal boot, drop to an EFI shell, then ran the following
# which will load the PK/Kek1 and Microsoft keys and enable SecureBoot
Shell> fs0:
FS0:\> EnrollDefaultKeys.efi
info: SetupMode=1 SecureBoot=0 SecureBootEnable=0 CustomMode=0 VendorKeys=1
info: SetupMode=0 SecureBoot=1 SecureBootEnable=1 CustomMode=0 VendorKeys=0
info: success
FS0:\> reset -s
# Then, finally, try and boot in SB mode:
virsh start focal; virsh console focal

[Dimitri John Ledkov (xnox)]

le sigh

[dann frazier (dannf)]

I dug up my arm64/disco VM that I had hacked up to test this shim build before MS had signed it:

ubuntu@disco:~$ sudo mokutil --sb-state
SecureBoot enabled

So what's the relevant difference between this working config and the broken one?

Looks like I had set it up following:
  https://wiki.archlinux.org/index.php/Secure_Boot

That is, I had created and installed unique PK, KEK & db keys:
ubuntu@disco:~$ sudo mokutil --pk | grep Issuer
        Issuer: CN=my Platform Key
ubuntu@disco:~$ sudo mokutil --kek | grep Issuer
        Issuer: CN=my Key Exchange Key
ubuntu@disco:~$ sudo mokutil --db | grep Issuer
        Issuer: CN=my Signature Database key
ubuntu@disco:~$ sudo mokutil --dbx | grep Issuer
ubuntu@disco:~$

I had signed shim w/ my custom db key:
ubuntu@disco:~$ sudo sbverify --cert db.crt /boot/efi/EFI/ubuntu/shimaa64.efi
warning: data remaining[836920 vs 900344]: gaps between PE/COFF sections?
Signature verification OK

And apparently GRUB as well:
ubuntu@disco:~$ sudo sbverify --cert db.crt /boot/efi/EFI/ubuntu/grubaa64.efi
Signature verification OK

While the kernel is an unmodified signed Canonical image.

Some package versions:
ubuntu@disco:~$ dpkg -l | grep -e shim
ii shim 15+1552672080.a4a1fbe-0ubuntu1 arm64 boot loader to chain-load signed boot loaders under Secure Boot
ii shim-signed 1.40~uefi1+dannf.1+15+1552672080.a4a1fbe-0ubuntu1 arm64 Secure Boot chain-loading bootloader (Microsoft-signed binary)
ubuntu@disco:~$ dpkg -l | grep grub
ii grub-common 2.02+dfsg1-12ubuntu2.1 arm64 GRand Unified Bootloader (common files)
ii grub-efi-arm64 2.02+dfsg1-12ubuntu2.1 arm64 GRand Unified Bootloader, version 2 (ARM64 UEFI version)
ii grub-efi-arm64-bin 2.02+dfsg1-12ubuntu2.1 arm64 GRand Unified Bootloader, version 2 (ARM64 UEFI modules)
ii grub-efi-arm64-signed 1.115+2.02+dfsg1-12ubuntu2 arm64 GRand Unified Bootloader, version 2 (EFI-ARM64 version, signed)
ii grub2-common 2.02+dfsg1-12ubuntu2.1 arm64 GRand Unified Bootloader (common files for version 2)
ubuntu@disco:~$ dpkg -l | grep linux-image
ii linux-image-5.0.0-36-generic 5.0.0-36.39 arm64 Signed kernel image generic
ii linux-image-5.0.0-37-generic 5.0.0-37.40 arm64 Signed kernel image generic
ii linux-image-virtual 5.0.0.37.39 arm64 Virtual Linux kernel image

And from the host:
ii qemu-efi-aarch64 0~20191122.bd85bf54-1 all UEFI firmware for 64-bit ARM virtual machines

[dann frazier (dannf)]

It does not appear to be an issue with the signed kernel - the same one that fails to build in my focal/MS-signed VM does boot when installed in the disco/self-signed VM.

[dann frazier (dannf)]

The issue seems to be related to GRUB. If I upgrade my working-in-SB-disco VM to focal's grub-efi-arm64-signed, it now fails. If I downgrade my *not*-working-in-SB-focal VM to disco's grub-efi-arm64-signed, it now boots in SB mode.

[dann frazier (dannf)]

Upstream GRUB 2.02 also fails - so there's likely something in the disco patchset that is required.

[Julian Andres Klode (juliank)]

Does it work with new grub and old shim, though? As we think new shim is somewhat buggy, it also fails to load fwupd on x86.

[dann frazier (dannf)]

@juliank: The shim build is the same in both cases. The only difference is that one is signed by MS, the other self-signed. I did notice that key enrollment didn't work in my guest, which maybe a failure to execute mma64.efi, and therefore maybe related to bug 1864223.

My hypothesis is that the SB support for arm64 is not upstream, and there was a regression introduced when rebasing those patches on GRUB 2.04.

[Steve Langasek (vorlon)]

also note that the previous version of shim did not work on arm64 AT ALL. This version of shim we just got signed is the first one with arm64 support.

[dann frazier (dannf)]

I've got a hacked together focal-based GRUB I created by replacing some patches w/ updated versions from the rhboot repository. It still need quite a bit of cleanup though.

[dann frazier (dannf)]

I've compared the list of patches squashed into the ubuntu-linuxefi.patch we have in focal with the current set of patches in the fedora-32 branch of https://github.com/rhboot/grub2, which is the upstream for our SB patches AIUI. There has been some rework of the linuxefi command since the version we last picked up - but it does appear to have equivalent functionality and fixes this regression. My proposal would be to do a "replace and rebase" of those patches to fix this bug and, presumably simplify focal maintenance somewhat by being more in sync with our upstream.

[dann frazier (dannf)]

I've nearly completed the rebase - here's what I have so far:
  https://git.launchpad.net/~dannf/grub/log/?h=secureboot-rebase

However, it fails to build on armhf now due to the changes in how the fdt module is built:
  https://launchpad.net/~dannf/+archive/ubuntu/test/+build/18793364
(That's a log from a slightly older version of my rebase, but the armhf failure is the same)

I need to find some doc or human that can help me understand the wizardry of Makefile.core.def to figure out how to fix that.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert Browne (rob8888)]

Upgraded grub-efi-amd64 to 2.04-1ubuntu26.2
4 kernels and recovery in menu none boot
So also effects AMD64
Disable secureboot allows kernel to boot

[Julian Andres Klode (juliank)]

Robert Browne (rob8888): Please open a separate issue with ubuntu-bug grub-efi-amd64-signed

[Robert Browne (rob8888)]

See ubuntu-bug 1890983

[Alfonso Sanchez-Beato (alfonsosanchezbeato)]

After working on LP: #1892770, I think that this is the same problem. grub in focal/arm64 is trying to use EFI boot services to start the kernel, after having verified the kernel with the shim. EFI does not have access to the cert in the shim so it refuses to load the kernel. There is a workaround which is adding Canonical Master key to the EFI DB, so EFI can actually verify the kernel.

Looking at "loader/arm64/linux.c" in the sources seems to validate this. In focal [1], grub_arch_efi_linux_boot_image() calls EFI boot services b->load_image() and b->start_image() which is the same in upstream code.

However, in disco [2] the EFI boot services are not used - the sequence is grub_armxx_efi_linux_boot_image() -> grub_efi_linux_boot() -> calls directly an address in the loaded kernel.

The patch that makes the change in disco is debian/patches/linuxefi_load_arm_with_sb.patch (https://github.com/rhboot/grub2/commit/2786ab864cf00c15123320671f653e9a36ba12b4).

[1] https://git.launchpad.net/ubuntu/+source/grub2/tree/grub-core/loader/arm64/linux.c?h=applied/ubuntu/focal-updates#n122
[2] https://git.launchpad.net/ubuntu/+source/grub2/tree/grub-core/loader/arm64/linux.c?h=applied/ubuntu/disco-updates#n157

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu34

---------------
grub2 (2.04-1ubuntu34) groovy; urgency=medium

  * configure.ac: one more dejavu font search path

 -- Dimitri John Ledkov <email address hidden> Mon, 14 Sep 2020 10:53:07 +0100

[Łukasz Zemczak (sil2100)]

Hello dann, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Julian Andres Klode (juliank)]

WIP, current state:

$ sudo dpkg -l grub-efi-arm64-signed shim-signed
[...]
ii grub-efi-arm64-signed 1.142.7+2.04-1ubuntu26.5 arm64 [...]
ii shim-signed 1.40.4+15+1552672080.a4a1fbe-0ubuntu2 arm64 [...]
ubuntu@instance-1:~$ sudo mokutil --sb-state
SecureBoot enabled
ubuntu@instance-1:~$ sudo dmesg | grep "secureboot"
[ 0.000000] secureboot: Secure boot enabled

4b) After installing unsigned kernel, it was rejected as unsigned correctly.

TODO: Verify that unsigned grub is correctly rejected.

[Julian Andres Klode (juliank)]

Finally ...

4a) Verify that it's broken after rollback of grub:

```
$ sudo eatmydata apt install ~i~ngrub/focal -V
[...]
The following packages will be DOWNGRADED:
   grub-common (2.04-1ubuntu26.5 => 2.04-1ubuntu26.4)
   grub-efi-arm64 (2.04-1ubuntu26.5 => 2.04-1ubuntu26.4)
   grub-efi-arm64-bin (2.04-1ubuntu26.5 => 2.04-1ubuntu26.4)
   grub-efi-arm64-signed (1.142.7+2.04-1ubuntu26.5 => 1.142.6+2.04-1ubuntu26.4)
   grub2-common (2.04-1ubuntu26.5 => 2.04-1ubuntu26.4)
[...]
```

Produces

```
BdsDxe: loading Boot0007 "ubuntu" from HD(15,GPT,6849AD57-3BC8-45BF-8BBD-33D9B49A4CCC,0x800,0x31801)/\EFI\ubuntu\shimaa64.efi
BdsDxe: starting Boot0007 "ubuntu" from HD(15,GPT,6849AD57-3BC8-45BF-8BBD-33D9B49A4CCC,0x800,0x31801)/\EFI\ubuntu\shimaa64.efi
error: no suitable video mode found.
error: cannot load image.

  Failed to boot both default and fallback entries.

Press any key to continue...
```

As expected.

[dann frazier (dannf)]

Marking verification-failed due to bug 1897819

[Łukasz Zemczak (sil2100)]

The version of grub2 in the proposed pocket of Focal that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

[Łukasz Zemczak (sil2100)]

The version of grub2-signed in the proposed pocket of Focal that was purported to fix this bug report has been removed because one or more bugs that were to be fixed by the upload have failed verification and been in this state for more than 10 days.

[Brian Murray (brian-murray)]

Hello dann, or anyone else affected,

Accepted grub2 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/grub2/2.04-1ubuntu26.6 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

[Ubuntu SRU Bot (ubuntu-sru-bot)]

All autopkgtests for the newly accepted grub2 (2.04-1ubuntu26.6) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

ubuntu-image/1.10+20.04ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#grub2

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

[Dimitri John Ledkov (xnox)]

setup the VM and enrolled keys.

Checked that grub live-server iso arm64 boots; and focal 20.04.1 does not.

Disabled secureboot, installed focal; upgraded to grub2 packages from proposed i.e. 2.04-1ubuntu26.6 & 1.142.8+2.04-1ubuntu26.6.

rebooted, enabled secureboot.

horay system boots correctly with secureboot, as said in the bootctl (secure boot enabled).

[Launchpad Janitor (janitor)]

This bug was fixed in the package grub2 - 2.04-1ubuntu26.6

---------------
grub2 (2.04-1ubuntu26.6) focal; urgency=medium

  * postinst.in, grub-multi-install: fix logic of skipping installing onto
    any device, if one chose to not install bootloader on any device. LP:
    #1896608
  * Do not finalize params twice on arm64. LP: #1897819

grub2 (2.04-1ubuntu26.5) focal; urgency=medium

  * ubuntu-linuxefi-arm64.patch: Fix build on armhf (LP: #1862279)

 -- Dimitri John Ledkov <email address hidden> Thu, 01 Oct 2020 23:19:24 +0800

[Łukasz Zemczak (sil2100)]

The verification of the Stable Release Update for grub2 has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

[Dimitri John Ledkov (xnox)]

grub2-signed (1.142.8) focal; urgency=medium

  * Rebuild against grub2 2.04-1ubuntu26.6.

 -- Dimitri John Ledkov <email address hidden> Thu, 01 Oct 2020 23:30:47 +0800

Has been in focal-updates since 2020-10-26.