Booting a QEMU aarch64 VM using kernel+initrd results in an EFI Synchronous Exception

[Corey Melanson]

I am attempting to create an aarch64 image using QEMU and virt-install on an x86_64 Ubuntu 22.04 server.

Using qemu-efi-aarch64 versions 2022.02-3ubuntu0.22.04.1 or 2022.02-3 both fail to boot the kernel when using the --location option (so kernel args can be supplied).

I can get it to succeed by downgrading qemu-efi-aarch64 to 20191122.bd85bf54-2ubuntu3.4 from the repo for another Ubuntu release.

I also tried qemu-efi-aarch64 2023.05-1 and it boots to the UEFI shell not the Linux kernel.

This is the command to boot the aarch64 VM:
virt-install \
  --name ubuntu2204 \
  --arch aarch64 \
  --ram 4096 \
  --disk path=/images/ubuntu2204.raw,format=raw,driver_type=raw,bus=virtio \
  --vcpus 4 \
  --os-variant ubuntu22.04 \
  --network bridge=virbr0 \
  --graphics none \
  --console pty,target_type=serial \
  --location /images/ubuntu-22.04.3-live-server-arm64.iso,kernel=casper/vmlinuz,initrd=casper/initrd \
  --extra-args 'auto=true priority=critical console=ttyAMA0,115200n8 serial autoinstall' \
  --noreboot

This results in the following output:
Starting install...
Retrieving 'vmlinuz' | 0 B 00:00:00 ...
Retrieving 'initrd' | 89 MB 00:00:00 ...
Creating domain... | 0 B 00:00:00
Running text console command: virsh --connect qemu:///system console ubuntu2204
Connected to domain 'ubuntu2204'
Escape character is ^] (Ctrl + ])

BdsDxe: failed to load Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x1,0x4)/Pci(0x0,0x0): Not Found
BdsDxe: failed to load Boot0001 "UEFI Misc Device" from VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00): Not Found
BdsDxe: loading Boot0003 "EFI Internal Shell" from Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
BdsDxe: starting Boot0003 "EFI Internal Shell" from Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)

UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table

Synchronous Exception at 0x000000012F0D496C

Synchronous Exception at 0x000000012F0D496C

Using qemu-efi-aarch64 to 20191122.bd85bf54-2ubuntu3.4 I get the following output:

Starting install...
Retrieving 'vmlinuz' | 0 B 00:00:00 ...
Retrieving 'initrd' | 100 MB 00:00:00 ...
Creating domain... | 0 B 00:00:00
Running text console command: virsh --connect qemu:///system console ubuntu2204
Connected to domain 'ubuntu2204'
Escape character is ^] (Ctrl + ])

EFI stub: Booting Linux Kernel...
EFI stub: Generating empty DTB
EFI stub: Loaded initrd from command line option
EFI stub: Exiting boot services...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd070]
[ 0.000000] Linux version 5.15.0-78-generic (buildd@bos02-arm64-036) (gcc (Ubuntu 11.3.0-1ubuntu1~22.04.1) 11.3.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #85-Ubuntu SMP Fri Jul 7 15:29:30 UTC 2023 (Ubuntu 5.15.0-78.85-generic 5.15.99)
[ 0.000000] efi: EFI v2.70 by EDK II
.... etc

ProblemType: Bug
DistroRelease: Ubuntu 22.04
Package: qemu-efi-aarch64 2022.02-3ubuntu0.22.04.1
ProcVersionSignature: Ubuntu 5.15.0-56.62-generic 5.15.64
Uname: Linux 5.15.0-56-generic x86_64
ApportVersion: 2.20.11-0ubuntu82.4
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Fri Sep 22 14:05:38 2023
Dependencies:

PackageArchitecture: all
ProcEnviron:
 SHELL=/bin/bash
 LANG=C.UTF-8
 TERM=xterm-256color
 PATH=(custom, no user)
RebootRequiredPkgs: Error: path contained symlinks.
SourcePackage: edk2
UpgradeStatus: No upgrade log present (probably fresh install)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[dann frazier (dannf)]

Thank you for the quality bug report Corey. Sorry it took so long to get back to you, but I've never used this boot method, and I needed to find focused time to figure out what was going on.

It turns out that the relevant change that is impacting you is that we introduced secure-boot enabled images for AArch64 with the Microsoft keys pre-enrolled (/usr/share/AAVMF/AAVMF_*ms.fd) starting in 2021.02-1. virt-install/libvirt now choose the secure boot images by default because of the ordering of descriptors in /usr/share/qemu/firmware. Because you are now booting with secure boot active, the firmware is trying to verify the kernel with Microsoft's keys. Ubuntu kernels are not signed by Microsoft directly - you are required to boot our MS-signed shim first, which will add trust for Canonical-signed binaries. So, the firmware is aborting that boot option and falling back to the EFI Shell.

This is behaving as designed.

You can override this and restore the prior behavior by specifying the non-ms images on the virt-install command line:

--boot loader=/usr/share/AAVMF/AAVMF_CODE.fd,loader_ro=yes,loader_type=pflash

The 2nd part you noted here is that, once the firmware decides to abort the kernel boot and move on to the Shell, the jammy version of the firmware crashes w/ Synchronous Aborts. This was an upstream bug that was fixed in 2023.02 with this commit:

commit 592bf33a29360e8a447ed8b75daefe94c0888642
Author: Michael D Kinney <michael.d.kinney@XXX>
Date: Fri Dec 9 07:53:21 2022 -0800

    ShellPkg/Shell: Do not set end device path if already end

    Update Shell Protocol EfiShellGetMapFromDevicePath() to not
    set the end if the device path if it is already an end of
    entire device path. This removes a write operation that can
    cause failures if the Device Path Protocol is mapped to
    read-only memory. In general Device Path Protocols should not
    be modified unless the API explicitly states that the device
    path is modified.

That's why you see the crash in jammy, but not in 2023.05-1. Since this is the only behavior that is not working as designed, I suggest we limit the scope of this bug to the Shell crash, and mark it Fix Released for releases that have this fix.