GRUB refuses to boot a 32-bit kernel when in EFI mode

[Hamish McIntyre-Bhatty]

Also reported at https://savannah.gnu.org/bugs/index.php?58300, but doesn't occur in Debian's v2.04 from Bulleye, and only occurred in the last few months, so it might be an Ubuntu bug.

This is grub version 2.02-2ubuntu8.15 as reported by "apt show grub-efi"

GRUB2 fails to boot a 32-bit kernel when started in EFI mode (64-bit EFI) on a 64-bit x86 CPU, and gives the message:

"error: kernel doesn't support 64-bit CPUs"

However, when a bios grub image made by the same version of grub is used, with the same kernel, on the same CPU, everything is normal and the kernel boots as expected.

Hence, I know this kernel will boot on a 64-bit CPU, and with a previous version of GRUB 2 (unfortunately I don't know which version), it also booted fine in 64-bit mode using GRUB-EFI.

Running with debug=all doesn't seem to provide any extra useful information, as far as I can tell - it just lists sectors being read and then freed.

Any ideas as to what's going on?

[Alkis Georgopoulos (alkisg)]

This problem is Ubuntu-specific, caused by that line:
https://git.launchpad.net/ubuntu/+source/grub2/tree/debian/patches/ubuntu-linuxefi.patch#n2105

That line was added by that commit:
https://git.launchpad.net/ubuntu/+source/grub2/commit/?id=6a814c759e10feafb40c3669be30aa51eb5ce39b

@cyphermox, this prohibits loading 32bit kernels under UEFI, would it be possible to revert it?
For example, before this patch, I was able to boot lubuntu-18.04-desktop-i386.iso under UEFI using https://github.com/alkisg/liveusb; after the patch, I can't.

True, the use cases for booting 32bit kernels under UEFI are limited, but is there a reason to have an Ubuntu-specific patch to block it when it works fine?

[Hamish McIntyre-Bhatty (hamishmb)]

Thanks for letting me know.

I'm not sure why that change is there either, but I guess there must be a reason.

[Alkis Georgopoulos (alkisg)]

Around 20 schools so far were affected by this, more will be affected in September.

This regression breaks existing installations: 32bit Ubuntu 18.04 in UEFI mode that just run `apt full-upgrade`, and they're no longer able to boot.

Could we please have some feedback? Even if it is "don't care; won't fix", we will know that we should solve it locally, e.g. via a PPA.

[Hamish McIntyre-Bhatty (hamishmb)]

Yes, it'd be good to see this fixed, even if it's not an official fix. I care more if it's affecting schools rather than just me with my weird setup.

For what it's worth, Debian Buster's packages don't seem to exhibit these issues - could install those and lock the version perhaps?

[Alkis Georgopoulos (alkisg)]

For secure boot to work, the distribution's grub is needed, as it contains the distribution's signing keys.

For now, as a workaround, we're using bionic's grub, for example:
https://github.com/alkisg/ltsp5-uefi/blob/master/ltsp5-uefi#L49

Let's hope we hear back from the Ubuntu developers...

Would a patch help? It's just about removing this line:
https://git.launchpad.net/ubuntu/+source/grub2/tree/debian/patches/ubuntu-linuxefi.patch#n2105

[Alkis Georgopoulos (alkisg)]

> we're using bionic's grub

I meant the original one from bionic and not from bionic-updates,
because the updated bionic grub is broken.

[Alkis Georgopoulos (alkisg)]

For those looking for the last good binaries, some notes.

Signed (support secure boot), need extraction from the .deb:
Good: https://launchpad.net/ubuntu/+source/grub2-signed/1.136/+build/18767810
Bad: https://launchpad.net/ubuntu/+source/grub2-signed/1.137/+build/18827799

Unsigned, but directly downloadable with plain wget:
Good: http://archive.ubuntu.com/ubuntu/dists/eoan/main/uefi/grub2-amd64/current/
Bad: http://archive.ubuntu.com/ubuntu/dists/eoan-updates/main/uefi/grub2-amd64/current/

Grub publishing history:
https://launchpad.net/ubuntu/+source/grub2-signed/+publishinghistory
Everything later than focal 1.137 and eoan 1.128.2 are broken.

Main grub launchpad page:
https://launchpad.net/ubuntu/+source/grub2-signed

[markakis (grigoris-markakis)]

Hundreds (perhaps many more) computers in school labs in Greece are affected by this bug.

[pemartins (paulo76-algarve)]

This bug stops Android-x86 32 bits systems from booting in 64 bits machines.
The 32 bits Android-x86 and Android-x86 based OS (like PhoenixOS, RemixOS, PrimeOS and others) are needed for 64 bit cpus that do not have SSE4.1 and SSE4.2, i.e. dual core cpus.

This is a very serious bug, it literally stops operating systems from booting. Please fix this asap.

[Hamish McIntyre-Bhatty (hamishmb)]

That might be unrelated, unless they use Ubuntu grub packages/patches?

[Julian Andres Klode (juliank)]

FWIW, to address some comments:

> This problem is Ubuntu-specific

It's not. It comes from https://github.com/rhboot/grub2/commit/1c88c700148acf02863a350055a43eb87e16bbe5 - which adds support for loading 64-bit kernels on 32-bit UEFIs.

> For now, as a workaround, we're using bionic's grub, for example: https://github.com/alkisg/ltsp5-uefi/blob/master/ltsp5-uefi#L49

Why not use a 64-bit kernel? Should be a much better option.

> This bug stops Android-x86 32 bits systems from booting in 64 bits machines.

You can't boot them via Ubuntu's grub true, but then they probably have their own bootloader?

[Hamish McIntyre-Bhatty (hamishmb)]

I feel like this is maybe missing the point. These things seem clear to me:

1. Loading 64-bit kernels on 32-bit UEFIs is a useful feature to have and I'm glad it was added.

2. AFAICT, the patch was not intended to break loading 32-bit kernels on 64-bit systems (and also what would be the point of such a change?), but if it was, I don't understand the reasoning behind doing this.

3. Of course booting a 64-bit kernel would work, but why should we be pushing updates that break existing installations due to what seems to be either an unintended consequence, or a "computer says no" type error even though the kernel is perfectly capable of booting. The error is misleading at best.

As such, I feel like this issue still needs to be fixed.

[Julian Andres Klode (juliank)]

My understanding is that our installers do not support installing i386 Ubuntu on UEFI systems. We only support amd64 on UEFI. If you manually hacked together an i386 install in UEFI mode, it's up to you to maintain.

[Julian Andres Klode (juliank)]

Running a 32-bit kernel on 64-bit UEFI is not supported. The kernel might not be able to interoperate with the UEFI implementation because some addresses would be outside its address range, and it would hence react strangely.

[Alkis Georgopoulos (alkisg)]

> Why not use a 64-bit kernel? Should be a much better option.

Because I'm trying to boot the stock, unmodified lubuntu-18.04.5-desktop-i386.iso that was released yesterday. It has a 32bit kernel.

> If you manually hacked together an i386 install in UEFI mode, it's up to you to maintain.

It's just a custom grub.cfg. I believe that using the "loopback" command to boot a vanilla ubuntu .iso should be within the "normal, supported use of grub". People using grub should be allowed to ask for support even though the installer uses grub in a different way. Grub doesn't exist in the archive only to serve ubiquity.

Anyways, since the problem came from redhat, we should move the discussion there and the fix will eventually reach Ubuntu, thank you very much for that information!

[Hamish McIntyre-Bhatty (hamishmb)]

Alkis,

Do you want me to file this for RedHat, or are you/have you already done so?

[Alkis Georgopoulos (alkisg)]

Hello Hamish,

please do; as currently I'm developing a lot of software for schools that open in September, so for the next few months I won't have time to file this. Thank you!

[Julian Andres Klode (juliank)]

Please don't. This is spiralling a bit out of control. I talked to the patch author and he told me what I told you - running 32-bit kernels on 64-bit UEFI does not work corectly. Please run 64-bit OS on 64-bit UEFI or boot using CSM in BIOS mode.

[Dimitri John Ledkov (xnox)]

Given that 64bit UEFI is there, it means that 64bit kernel can run, despite using 32bit userspace otherwise.

Why are you using inferior 32bit kernel, with otherwise 32bit userspace?

Please install and use 64bit kernel in such cases, i.e.

sudo dpkg --add-architecture amd64
sudo apt-get update
sudo apt-get install linux-generic:amd64

"works fine" is not true at all, lots of low-level things in Userspace and kernel are broken when one tries to force booting incompatible kernel due to different memory layout expectations. If you really want to fix it, it will need to go upstream to linux kernel to implement 64bit UEFI memory layouts in the 32bit kernel flavour.

[Dimitri John Ledkov (xnox)]

> Because I'm trying to boot the stock, unmodified lubuntu-18.04.5-desktop-i386.iso that was released yesterday. It has a 32bit kernel.

Why are you not using -amd64.iso?

[Dimitri John Ledkov (xnox)]

It was never our intention to support 64bit UEFI with 32bit kernel. Just because it seemed to work, whilst having lots of things broken at runtime, was never our intention to have supported.

[Hamish McIntyre-Bhatty (hamishmb)]

Okay then, I guess I won't report. 32-bit x86 is going away for a lot of distros so I doubt the kernel people will have any interest in doing it tbh.

Glad I'm moving my Debian live disk away from doing this then if it breaks lots of stuff :)

[Alkis Georgopoulos (alkisg)]

Thank for you all the feedback guys,

OK, since upstream already replied "won't fix", of course there's no point to report an issue.
As long as i386 installations are still supported, we that need this functionality, can keep using the last working grub.

> Why are you not using -amd64.iso?

One use case is to be able to use the same .iso or live USB or netbooted installation for many client computers. For example, we have a few hundred schools here that have both 32bit and 64bit clients; they all netboot from a 32bit installation so that the teacher doesn't have to maintain many installations. We can keep using the old grub until 18.04 is EOLed in 2023.

Another use case is to be able to troubleshoot 32bit-only bugs in 64bit hardware (that in some cases only supports UEFI booting); for example, Ryzen/amdgpu-based computers currently show black screen with 32bit installations yet they work fine with 64bit.

[pemartins (paulo76-algarve)]

I really thought, because I've read and heard so many times about it, that the Linux world was about freedom. Well, can't really say that someone imposing that other people were from now on prohibited from running 32 bits operating systems in an updated 64 bits grub/uefi/whatever suits that...

I've been running on my laptop, for the last many years, 32 bits Android-x86 based operating systems because the 64 bits ones won't work due to my cpu lacking sse 4.1 and 4.2. But thanks to the awesome Linux freedom, I can now choose not to do it anymore... you know, because I can't do it anymore.
No worries, at least this is great for having a laugh!

[Hamish McIntyre-Bhatty (hamishmb)]

As discussed I suppose you can either use an older version of GRUB-EFI (which as pointed out may cause unknown problems), or perhaps you can boot in CSM mode?

[Hamish McIntyre-Bhatty (hamishmb)]

NB: In Debian testing (11) my Ryzen 3600 system boots and runs a 32-bit image okay.

[Alkis Georgopoulos (alkisg)]

In case anyone needs to install the last known binaries that work, I uploaded a nice overview of the problem and direct links to the Ubuntu packages here:

https://github.com/alkisg/liveusb/issues/3