Comment 8 for bug 1895131

Hi Thomas. Glad to see you onboard here as well. ;)

First of all, I guess I should point out that UEFI bootloader boot from non ESP partitions is hardly a "trick". It is de-facto. That behaviour comes from the EDK2 source itself and, as my recent dealings with EDK2 indicate (in [1]) that the EDK2/UEFI folks are more inclined to have the existing EDK2 code drive the specs, especially for boot, than the opposite. Which means that, if the default firmware you get from EDK2 pretty much sees any /efi/boot/boot###.efi bootloader that resides on a non-ESP partition it has a file system for as something it should provide as a boot option, then this is as established as if it was part of the specs. Hence "de facto".

So I see little point in trying to stir this discussion into asking whether distros may be technically entitled not to bother with such a method of creating UEFI boot media as:
1. People *ARE* going to be attempting to use these methods, since they will be supported by their platform's UEFI firmware, regardless of whether they are explicitly backed up by the specs or not.
2. Even if General Data partition boot was to be formally outlawed tomorrow, it's not that difficult to change the partition type to ESP when creating the media in the manner I highlighted, and then we're back to square one, with missing EFI bootloaders on the ISO-9660 file system preventing boot by simply extracting the content to an ESP partition.

So since you provided a comprehensive explanation (thanks!) as to why distros took it to moving to using a boot/grub/efi.img for the EFI bootloaders (I see where you're coming from now, but it's a shame you didn't consider the need for folks to be able to create bootable media by simply extracting the full ISO-9660 content onto their own data partition) then let me provide my Linux ISO creation requirements, RFC style:

For the purpose of providing a method of creation of UEFI boot installation media by simply enabling the user to extract the ISO content onto an existing data partition, a Linux installation ISO:

1. MUST NOT use symbolic links, for the file systems that it implements on the ISO (through the "hybrid" part), for any critical part of the boot or installation process. In other words, while one may create a symbolic '/doc/' link at the root level that points to something like '/release/documentation/', it should not use a symbolic link for vmlinuz, initrd or any of the installation packages, as these are unlikely to be preserved during ISO extraction onto an alternate file system.

2. MUST ensure that, along with ISO-9660, it supports FAT32 as a file system that can be used during installation (meaning that the default installation kernel should not forcibly remove vfat as a supported file system, and that the installation scripts should be able to handle the retrieval of content that resides on FAT file systems, and not just ISO-9660). Note that this point is conditional to not requiring the use of any file that larger than 4 GB. If that cannot be achieved then the next SHOULD becomes a MUST.

3. SHOULD ensure that, along with ISO-9660, it supports exFAT as a file system that can be used during installation. The obvious reasoning behind this has to do with the 4 GB limitation of FAT32 when it comes to individual files, which, thankfully, mostly due to the use of squashfs, is not yet a limit that is being encountered by distro installers, but that it makes sense to plan for. With Microsoft having recently granted "free use" of exFAT for the Linux kernel, it makes sense to foster the use of exFAT over alternatives like NTFS, as exFAT has also the advantage of being natively supported on MacOS.

4. MUST ensure that UEFI bootloaders, that enable the creation of bootable installation media by extracting the ISO-9660 content straight into a FAT32/exFAT, are present on the ISO-9660 file system in their expected location ('/efi/boot/boot####.efi').

If the above is respected, and right now Ubuntu only needs to sort out point 4, then the simplest and, I will assert, safest method of creating a UEFI bootable installation media can be expected to work.

It should be noted that a direct corollary of points 2. and 3. is that the naming scheme used for the ISO-9660 content needs to be more restrictive than what Rock-Ridge or Joliet allows, as, for instance, case sensitivity and special characters have to carefully considered, else a file name lookup that might work in an ISO-9660 environment might be broken for content that was extracted to FAT32/exFAT.

Also, whereas the exFAT push might seems a bit pointless for now, since the EDK2 doesn't provide a native exFAT driver, it should be noted that there does exist methods to load the required exFAT UEFI driver and then automatically chain load the EFI bootloader from the exFAT partition (I'm actually using such a method in Rufus, by creating a small 512KB "ESP" at the end of the drive [2], though right now this is mostly used for NTFS, which Ubuntu actually was supporting just fine last time I tested) and, unless someone does it before me, I do have some medium-term plan of bringing an exFAT driver proposal to EDK2. Once I do that, and if integrated, we should start to see UEFI platforms with native exFAT boot support, rendering the need to an additional hack for > 4 GB file support obsolete.

I'm hoping that the proposal highlighted above makes sense to all the parties involved.

Regards,

/Pete

[1] https://bugzilla.tianocore.org/show_bug.cgi?id=2831#c14
[2] https://github.com/pbatard/uefi-ntfs (Don't let the name fool you, this also supports exFAT)