Grub multiboot is unable to load Xen under EFI

[Dario Bertini]

As a new Xen user, when I install the `xen-hypervisor-amd64` package, I'd expect it to work fine.

But upon rebooting, even if Xen succeed to boot, the system is basically unusable, and there are plenty of Irq errors in dmesg.

It turns out that EFI is not very well supported under Xen:

http://wiki.xenproject.org/wiki/Xen_EFI

The alternatives are:

- boot directly from the Efi (instead of relying on grub's multiboot)
- enable legacy boot in the EFI, and reinstall grub in bios-mode (and hope that everything goes well)

Since I don't think that the latter can be automated, the first one would be a better approach for ubuntu's packaging, I think.

But even if that's not possible, or there are no resources for it, at least (until EFI on Grub won't be properly supported), either at install time or at boot time, the fact that the system is running on EFI should be detected, and explicitly warn the user (halt the boot process?) to avoid leaving him in a puzzlingly half-working system.

ProblemType: Bug
DistroRelease: Ubuntu 15.10
Package: xen-hypervisor-4.5-amd64 4.5.1-0ubuntu1.1
ProcVersionSignature: Ubuntu 4.2.0-18.22-generic 4.2.3
Uname: Linux 4.2.0-18-generic x86_64
ApportVersion: 2.19.1-0ubuntu5
Architecture: amd64
CurrentDesktop: Unity
Date: Sun Nov 29 19:19:05 2015
InstallationDate: Installed on 2015-11-22 (7 days ago)
InstallationMedia: Ubuntu 15.10 "Wily Werewolf" - Release amd64 (20151021)
SourcePackage: xen
UpgradeStatus: No upgrade log present (probably fresh install)
mtime.conffile..etc.default.grub.d.xen.cfg: 2015-11-24T13:27:42.307217

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Jeff Lane  (bladernr)]

I have seen this issue as well, after being pointed to problems booting Xen w/ Ubuntu by a colleague.

To recreate this I did the following:

Deployed a server in UEFI mode via MAAS 1.9 using the latest Xenial images from the MAAS daily image stream.
After deployment, SSH'd to the server and installed xen-hypervisor-4.6-amd64 via apt-get.
Rebooted system after installation of Xen.

The system then attempted to EFI PXE (set to do so by default). It successfully PXEd and the MAAS server directed it to boot locally. Then it attempted to boot the Xen Hypervisor and halted at loading the initial ramdisk.

At that point, the system abended and rebooted. It was completely unable to boot Xen.

[Stefan Bader (smb)]

My few cents here. Its unfortunate but its known and not simple to fix. The current multiboot protocol does not allow UEFI boot as things transition into a mode that shuts down certain services from the UEFI BIOS side. Which I think will cause the hypervisor not finding any memory since this is reported in a different way which is only available if the UEFI boot services were not stopped. Upstream was working on the problem on and off but I don't think there is a final solution yet.
There is a EFI executable added to /boot when installing the hypervisor but that would have to be loaded by the UEFI BIOS directly (not sure how passing command line arguments is supposed to be done exactly). It is not signed either, so secure boot would not work any way.
The best way forward IMO would be to have the needed fixes in multiboot2, so there is a way to boot via grub2 in UEFI mode. Still that has the secure boot topic unresolved.

Preventing installation of Xen when in UEFI mode is probably bad either (plus I am not sure it can be safely done). So maybe the best option is to clearly state in the release notes that Xen does currently not boot in UEFI mode.

[Dario Bertini (berdario)]

Thank you, anyhow concerning the last point:

> Preventing installation of Xen when in UEFI mode is probably bad either

Yes, but the release notes cannot be seen anywhere at installation time, if I'm not wrong. People would need to explicitly look for it.

Wouldn't it be possible to add a dpkg configure prompt ask for confirmation to the user? (just like for the accept-license prompts)

"We detected that you're running in UEFI mode, XEN is currently unsupported on this configuration. Continue at your own risk [yes] [abort]"

[Stefan Bader (smb)]

This might be nice to have but I would not see any reason to rush this. Xen is not on the ISO images (and the whole Xen host/hypervisor parts are in universe which usually means a lower priority on being perfect) so this can be fixed later, still.

Also a critical bug imo needs to be one that causes data loss (even high would not be true as that would be service disruptions) and this is an issue which affects new installations only. And it is recoverable at least using the recovery option of the server ISO. Should also be possible by accessing the grub menu and switching to the non-xen boot but I admit I would not be quick enough to hit left-shift during that little time that the hidden grub default allows.

For those reasons I would lower the priority to medium and add a task to add something to the release notes.

[Fake Name (lemuix-2)]

What part of this is "fixed"?

Anyways, xen can apparently support efi-booting as a efi executable directly, but for some reason ubuntu doesn't ship the "xen.efi" (see https://wiki.xenproject.org/wiki/Xen_EFI)?

While I understand that fixing this is kind of out of the scope of the ubuntu packager people, at least mark it "wontfix" or something.

[Stefan Bader (smb)]

The release notes task is marked fixed (meaning the issue is mentioned there). Ubuntu does ship the EFI executable (see /boot/xen-<version>-amd64.efi. Of course its not simple to use it.
And lastly the bug is open as a reminder to follow development that tries to extend grub2 multiboot (multiboot2) to allow Xen being booted via grub on EFI systems, too. Which would be the better way to integrate into any distro.

[Danny (dannyp777)]

I tried to use Xen with Ubuntu Noble Numbat 24.04 with UEFI SecureBoot and it fails on line 179 of sb.c : https://git.savannah.gnu.org/cgit/grub.git/tree/grub-core/kern/efi/sb.c with the message "prohibited by secure boot policy". I have repeated this with a clean Ubuntu Minimal Server install in a Qemu VM with UEFI. I am in the process of trying it with a clean Ubuntu Jammy minimal server VM. (which uses grub v2.06 instead of v2.12)
I was looking at the code and wondering if it was something to do with the multiboot grub verifier, but I think I may have to install a debug version of grub to get the debug messages.
I haven't found any bugs on the main grub bug report page about it as yet (https://savannah.gnu.org/bugs/?group=grub)
I haven't submitted anything to the main bug list yet as I more familiar with launchpad.

[Danny (dannyp777)]

The problem is in Ubuntu Jammy with version v2.06 of grub as well.

[Danny (dannyp777)]

The problem is in Fedora 40 too.
I have also tried the following (with Ubuntu Noble Numbat 24.04):
Added new CodeSigning key to UEFI db keystore.
Rebuilt shim-15.8 with the new key.
Added Canonical Codesigning key from shim-signed source package to UEFI db keystore.
Signed all my efi's with the new key (shimx64, mmx64, fbx64, grubx64, Xen)
I didn't sign the kernel as I added the Canonical key to the db keystore.
Results:
Booted linux kernel fine.
Still get same problem with Xen even though its signed, and the key is being recognised by SecureBoot and shim on the other efi's.
Also added `insmod multiboot2` statements to `/etc/grub.d/20_linux_xen` so they would be at the top of each xen menu block in the `/boot/efi/grub/grub.cfg`

Maybe I should get on the grub mailing lists before spending more time on this in case I'm wasting my time.

[Danny (dannyp777)]

I see according to this: `https://xenbits.xen.org/docs/unstable/support-matrix.html` that Host UEFI SecureBoot status is 'Experimental' on versions 4.18 and 4.19, and no status on previous versions. Maybe I'll try v4.18 (Ubuntu Noble Numbat 24.04 uses Xen v4.17)

[Sergio Durigan Junior (sergiodj)]

Hello Danny,

Thanks for the initial investigation. I haven't looked into the problem, but if this is marked as an experimental feature on recent upstream versions, it is unlikely that it will make it into Noble's xen package. It's already a challenge to justify backporting a feature; experimental code is even harder.

I'll defer to someone with more expertise on the subject to weigh in here, though.