Groovy Desktop *BREAKS* the most common method of creating UEFI bootable drives for Ubuntu installation

[Akeo]

As opposed to 20.04, Groovy Desktop decided to do away with the EFI boot files that used to reside in `/EFI/BOOT/` on the ISO, and instead moved them away into a FAT image located at /boot/grub/efi.img.

While this works fine when writing the image in DD mode, it completely **BREAKS** the common method, used by many many users on Windows, MacOS and other platforms, of simply formatting a USB drive to FAT32 and then copying the whole ISO content there in order to create a UEFI bootable Ubuntu installation media.

Please bear in mind that this method of creating UEFI boot media is favoured by many on account that:
* It doesn't require the installation of third party software like balenaEtcher or Rufus to create the media, especially on Windows
* It is less risky to use than dd, on account that it's less prone to making a mistake with regards to the target disk. Especially, not everyone has access to 'dd', or is familiar enough with it, or even wants to use it if there exists an alternative that lets them access the content of their drive (e.g. on Windows).
* It leverages the *NATIVE* ability of UEFI to pick a bootloader from /EFI/BOOT/ which was introduced precisely to make the creation of bootable media through third party utilities (including dd) a thing of the past.

So, let me start by giving a stern warning here:

UBUNTU PEOPLE, BY HIDING THE EFI BOOT FILES AWAY IN THE ISO, YOU ARE **NOT** HELPING YOUR USERS. INSTEAD, YOU ARE ACTIVELY **DEGRADING** THE USER INSTALLATION EXPERIENCE. PLEASE DON'T DO THAT!

My questions therefore are twofold:

1. What on earth was the rationale behind this move? What exactly is to be gained here?? Ubuntu 20.4 was perfectly fine with the GRUB boot files in /EFI/BOOT/ on the ISO file system structure, and, as pointed out above, it's hard to see how hiding these files away in efi.img is going to improve user experience when this breaks the simplest method of creation of a UEFI bootable media. So what prompted this sudden unwarranted change, and why didn't anybody realize that this would make the Ubuntu media creation experience subpar in terms of UEFI install?

2. Can this change please, please, **PLEASE**, be reverted? I know that drinking the ISOHybrid kool-aid and putting your eggs into one basket by declaring that `dd` is now the "one true way" of creating UEFI bootable media is very seducing from a maintainer's perspective. But don't remove features that helped foster the image of Ubuntu being focused on user-friendliness, and that are **ACTUALLY** used by more people than you realize. Else you may find that a move that actively prevents people from installing Linux in a manner they've been using for years, and that really has no reason to be broken because it's what UEFI was designed for, will be percieved as a **STRONG INDICATION** that Ubuntu is no longer caring about its users...

Thank you.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[C.S.Cameron (cscameron)]

I have been trying to boot the latest Groovy daily by loop mounting the ISO file. It works in BIOS mode but not in UEFI mode. This is worse than the problems with GRUB 2.04.

I am wondering if this is related to Bug https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1892754

[sudodus (nio-wiklund)]

@Akeo,

I have seen the problem that you address in this bug report, and I was afraid that it may cause problems for you[r tool] and others who use extracting methods. Thanks for speaking out load about it.

There is an additional problem too, that some computers do not even boot from cloned drives in UEFI mode (See https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148/comments/65 and the following comments).

@C.S.Cameron,

I have managed to loop mount the latest Groovy daily iso files (both in BIOS mode and UEFI mode) and make it boot. But it seems that I have not made that process work in all possible cases yet ...

[Michael Hudson-Doyle (mwhudson)]

I think this change https://code.launchpad.net/~mwhudson/debian-cd/half-the-grubs/+merge/387085 will make things more like they were.

[sudodus (nio-wiklund)]

@ Michael Hudson-Doyle (mwhudson),

Please let us know, when this change will be available for iso testing.

[Thomas Schmitt (scdbackup)]

Hi,

i am the developer of xorriso. If there are problems after changing
the xorriso arguments i am ready to help identifying the cause.

My own opinion about how Debain (and Ubuntu) ISOs for BIOS|EFI from CD|HDD
should look like is to see in
  https://lists.debian.org/debian-cd/2019/07/msg00007.html
(It uses ISOLINUX for BIOS, but could well use GRUB software with the
appropriate xorriso options for GRUB MBR code. Its main theme is to abandon
the hacky layout with invalid GPT that was introduced by Fedora in 2012.)

This opinion is strictly oriented towards a single ISO that is flatly
copied to its storage medium. The trick (*) with just extracting the ISO
into a FAT filesystem is not generically supported by my proposals.
But of course it would be possible to have an unpacked copy of the
EFI System Partition in ISO directory /EFI/BOOT, so that the trick works
again.

(*) It is not specified in UEFI which demands that the EFI partition has
to be marked by type 0xEF in MBR table, or Type GUID
C12A7328-F81F-11D2-BA4B-00A0C93EC93B in GPT. But Microsoft Inc. obviously
supports this trick and thus firmware programmers will hardly disable it.

------------------------------------------------------------------------

My advise to developers of ISO post-processing software is to join forces
and to write down a specification about what they expect from a bootable
ISO which shall be subject to their activities.

I am ready to participate in discussions about such a specification.

Have a nice day :)

Thomas

[sudodus (nio-wiklund)]

@ Thomas Schmitt (scdbackup),

Thanks for chipping in and offering help :-)

[Akeo (pbatard)]

Hi Thomas. Glad to see you onboard here as well. ;)

First of all, I guess I should point out that UEFI bootloader boot from non ESP partitions is hardly a "trick". It is de-facto. That behaviour comes from the EDK2 source itself and, as my recent dealings with EDK2 indicate (in [1]) that the EDK2/UEFI folks are more inclined to have the existing EDK2 code drive the specs, especially for boot, than the opposite. Which means that, if the default firmware you get from EDK2 pretty much sees any /efi/boot/boot###.efi bootloader that resides on a non-ESP partition it has a file system for as something it should provide as a boot option, then this is as established as if it was part of the specs. Hence "de facto".

So I see little point in trying to stir this discussion into asking whether distros may be technically entitled not to bother with such a method of creating UEFI boot media as:
1. People *ARE* going to be attempting to use these methods, since they will be supported by their platform's UEFI firmware, regardless of whether they are explicitly backed up by the specs or not.
2. Even if General Data partition boot was to be formally outlawed tomorrow, it's not that difficult to change the partition type to ESP when creating the media in the manner I highlighted, and then we're back to square one, with missing EFI bootloaders on the ISO-9660 file system preventing boot by simply extracting the content to an ESP partition.

So since you provided a comprehensive explanation (thanks!) as to why distros took it to moving to using a boot/grub/efi.img for the EFI bootloaders (I see where you're coming from now, but it's a shame you didn't consider the need for folks to be able to create bootable media by simply extracting the full ISO-9660 content onto their own data partition) then let me provide my Linux ISO creation requirements, RFC style:

For the purpose of providing a method of creation of UEFI boot installation media by simply enabling the user to extract the ISO content onto an existing data partition, a Linux installation ISO:

1. MUST NOT use symbolic links, for the file systems that it implements on the ISO (through the "hybrid" part), for any critical part of the boot or installation process. In other words, while one may create a symbolic '/doc/' link at the root level that points to something like '/release/documentation/', it should not use a symbolic link for vmlinuz, initrd or any of the installation packages, as these are unlikely to be preserved during ISO extraction onto an alternate file system.

2. MUST ensure that, along with ISO-9660, it supports FAT32 as a file system that can be used during installation (meaning that the default installation kernel should not forcibly remove vfat as a supported file system, and that the installation scripts should be able to handle the retrieval of content that resides on FAT file systems, and not just ISO-9660). Note that this point is conditional to not requiring the use of any file that larger than 4 GB. If that cannot be achieved then the next SHOULD becomes a MUST.

3. SHOULD ensure that, along with ISO-9660, it supports exFAT as a file system that can be used during installation. Th...

[Thomas Schmitt (scdbackup)]

Hi,

> the EDK2/UEFI folks are more inclined to have the existing EDK2 code
> drive the specs, especially for boot, than the opposite.

It really was supposed to be the other way round this time. :))
But well, firmware is as firmware is.

As said, having a copy of the ESP tree in the ISO should be easy to
achieve and not occupy too much space, except on CD sized images, where
every MB matters.

> the naming scheme used for the ISO-9660 content needs to be more
> restrictive than what Rock-Ridge or Joliet allows, as, for instance,
> case sensitivity and special characters have to carefully considered,
> else a file name lookup that might work in an ISO-9660 environment might
> be broken for content that was extracted to FAT32/exFAT.

That might be a show stopper with Debian-ish distros. They have long package
names and there is no guarantee that filenames truncated to 8.3 are still
unique.

Mounting ubuntu-19.04-desktop-amd64.iso and doing

  for i in $(find /mnt/iso/pool) ; do basename "$i" ; done | sort | less

i get to these problematic name clusters:

  grub-efi-amd64-bin_2.02+dfsg1-12ubuntu2_amd64.deb
  grub-efi-amd64-signed_1.115+2.02+dfsg1-12ubuntu2_amd64.deb
  grub-efi-amd64_2.02+dfsg1-12ubuntu2_amd64.deb

  oem-config-gtk_19.04.9_all.deb
  oem-config-slideshow-ubuntu_146_all.deb
  oem-config_19.04.9_all.deb

With a Debian 9 amd64 DVD-1 image it is much worse because of more packages.

Of whatever you can convince debian-cd or debian-live, the package names are
in the realm of the package maintainers.

Have a nice day :)

Thomas

[Akeo (pbatard)]

Just going to point out that I was NOT talking about limiting file names to 8.3.

FAT32/vfat does has LFN support, and the UEFI FAT driver supports LFN as well, so, NO, what I was talking about was not about suddenly asking distro maintainers to use 8.3 names.

It's only about case sensitivity (since FAT will see 'filename' and 'FileName' as the same thing) and avoiding characters like '*', '?', '<', '>', ':', '|'... That's all.

So, let's not framed this matter into something it was never about, thanks.

[Thomas Schmitt (scdbackup)]

Hi,

> avoiding characters like '*', '?', '<', '>', ':', '|'... That's all.

Except ':' the others have a meaning to the shell and thus would be a
controversial choice.i
In case of Debian it is nearly defined what characters are allowed in a
package name.

  https://www.debian.org/doc/manuals/maint-guide/first.en.html#namever
  "Package name and version [...]
   You must choose the package name to consist only of lower case letters
   (a-z), digits (0-9), plus (+) and minus (-) signs, and periods (.).
   It must be at least two characters long, must start with an
   alphanumeric character, [...]
   You should choose the upstream version to consist only of alphanumerics
   (0-9A-Za-z), plus signs (+), tildes (~), and periods (.)."

So the version part of the .deb file name might contain ':', although this
is discouraged.

Consider to give the complete list of bad characters with your points 2
and 3.

Have a nice day :)

Thomas

[Steve Langasek (vorlon)]

Thomas, thanks for weighing in. Reading https://lists.debian.org/debian-cd/2019/07/msg00007.html was very helpful in clarifying the options needed to get the image structure we are looking for. I think we may now have the xorriso options sorted out (though, this does not address the /EFI/BOOT consideration at this point).

[Thomas Schmitt (scdbackup)]

Hi,

Steve Langasek wrote:
> Reading https://lists.debian.org/debian-cd/2019/07/msg00007.html was very
> helpful in clarifyingthe options needed to get the image structure we
> are looking for.

At least the ISO would comply to the specs, although this is in no way
mandatory for boot success. :))

> this does not address the /EFI/BOOT consideration at this point

Akeo has a valid point with demanding support for the method of unpacking
the ISO content to a FAT filesystem. Not only is it less dangerous than
our flat image copy onto the whole USB stick, but it is also something
that MS-Windows users are used to.

The price for having a copy of the EFI partition tree in the ISO should
be bearable.

On the other hand, this won't work with legacy BIOS (at least not without
tricky post-processing of ISOLINUX) and it invites problems with filenames.

This brings me to a question about .deb file names.
I see in
  https://packages.debian.org/sid/allpackages?format=txt.gz
that many packages have version texts with ":". Like
  a2ps (1:4.14-5 [alpha, amd64, ...
But the .deb file of a2ps does not show the "1:" part:
  http://ftp.de.debian.org/debian/pool/main/a/a2ps/a2ps_4.14-5_amd64.deb

Being a mere sponsored uplader, i fail to find the description how the
version gets transmogrified before becomming part of the .deb file name.
If it is clear that no ":" can sneak in via version, this risk about FAT
filenames would be much reduced.
(There remains the risk of stubborn package maintainers, though.)

I understand from Akeo's initial protest that currently the filenames in
Ubuntu ISOs are not an obstacle for the copy method which he defends.
Nevertheless, it would be re-assuring if there was a rule or algorithm
which prevents ":" from getting into .deb names.

Have a nice day :)

Thomas

[Ubuntu QA Website (ubuntuqa)]

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
http://iso.qa.ubuntu.com/qatracker/reports/bugs/1895131

[Paride Legovini (paride)]

Hi,

This bug is affecting the automated ISO testing of the Desktop and Server images, as UTAH (the test framework) validates the EFI files contained in the main iso9660 filesystem as part of the ISO static validation stage. An actual install with UEFI boot is also performed as part of the test suite, so the EFI files actually used to boot are tested too.

I was thinking to modify UTAH so it does the EFI static validation on the actual ESP, however if the EFI copy in the iso9660 filesystem is going to be retained to support the creation of bootable drives by plain copy, as this bug asks, then we'll want to validate the copy.

In other words the outcome of this bug will influence if and how the ISO static validation is going to be fixed. For the moment we'll skip the EFI static validation stage, so the other tests can run.

[Akeo (pbatard)]

Any update on this?

As a reminder, ALL that is being asked from this bug is to duplicate the files and directories currently found in /boot/grub/efi.img to the root level of the ISO. That's it.

It's a very straightforward fix, and, with the beta now being out, the need to fix this becomes slightly more urgent, lest you want to let all the people who previously relied on the ability to simply extract the ISO files onto a FAT32 partition to create a UEFI bootable media down.

[sudodus (nio-wiklund)]

@ Akeo,

If I understand correctly, this bug is fixed in the current daily Groovy iso files. These files are newer than the Groovy beta iso files.

http://iso.qa.ubuntu.com/qatracker/milestones/413/builds

I can extract the content of an iso file to a FAT32 partition and get a USB boot drive, that boots in UEFI mode. Please try if it works for you too.

[Akeo (pbatard)]

@sudodus, you're right. I just tested the 2020.10.04 daily build and, as opposed to the beta, it does contain a /efi/ directory at root level with all the relevant files.

If this is carried out into the release, then, as far as I am concerned, this bug can be closed.

[Paride Legovini (paride)]

I can confirm the /efi directory is back, although with different capitalization than before:

Focal:

/EFI
/EFI/BOOT
/EFI/BOOT/BOOTx64.EFI
/EFI/BOOT/grubx64.efi
/EFI/BOOT/mmx64.efi

Groovy:

/efi/
/efi/boot
/efi/boot/bootx64.efi
/efi/boot/grubx64.efi
/efi/boot/mmx64.efi

This is a bit annoying from the static validation point of view, but can be worked around.

I can't spot what change caused the /efi directory to be included again in the ISO filesystem. Does anybody have a pointer?

[Michael Hudson-Doyle (mwhudson)]

On Tue, 6 Oct 2020 at 22:20, Paride Legovini <email address hidden>
wrote:

> I can't spot what change caused the /efi directory to be included again
> in the ISO filesystem. Does anybody have a pointer?

It was this upload I think:
https://launchpad.net/ubuntu/+source/cd-boot-images-amd64/7

[Paride Legovini (paride)]

Thanks! I keep discovering new ones :P

I'm marking this bug as Fix Released, as LP: #1886148 is.

[Thomas Schmitt (scdbackup)]

Hi,

> https://launchpad.net/ubuntu/+source/cd-boot-images-amd64/7

Oh. This explains why there is still an invalid GPT in
   http://cdimage.ubuntu.com/daily-live/current/groovy-desktop-amd64.iso
which i downloaded 2 days ago.

Above change has at its end:

+ echo 'xorriso -as mkisofs [...]
  -append_partition 2 0xef images/boot/grub/efi.img [...]
  -e --interval:appended_partition_2:all:: [...]
  -part_like_isohybrid -isohybrid-gpt-basdat [...]' > xorriso-cmd.txt

To blame for the invalid GPT is "-part_like_isohybrid -isohybrid-gpt-basdat".

------------------------------------------------------------------------

I downloaded a new groovy-desktop-amd64.iso and repacked it without these
options:

  dd if=groovy-desktop-amd64.iso bs=512 count=1 \
     of=groovy-desktop-amd64.mbr.img

  dd if=groovy-desktop-amd64.iso bs=512 skip=5715872 count=9952 \
     of=groovy-desktop-amd64.efi.img

  xorriso -as mkisofs \
    -o test.iso \
    -J -joliet-long -l \
    -b boot/grub/i386-pc/eltorito.img \
       -no-emul-boot -boot-load-size 4 -boot-info-table --grub2-boot-info \
    --grub2-mbr groovy-desktop-amd64.mbr.img \
    -append_partition 2 0xef groovy-desktop-amd64.efi.img \
    -eltorito-alt-boot \
    -e --interval:appended_partition_2:all:: \
       -no-emul-boot \
    -partition_offset 16 -r \
    /mnt/iso

The numbers 5715872 and 9952 can be obtained by "/sbin/fdisk -l" or by
xorriso inspection.

This inspection

  iso=groovy-desktop-amd64.iso

  xorriso -indev "$iso" -report_system_area plain -report_el_torito plain

yields for the downloaded ISO, iso=groovy-desktop-amd64.iso :

  Drive current: -indev 'groovy-desktop-amd64.iso'
  Media current: stdio file, overwriteable
  Media status : is written , is appendable
  Boot record : El Torito , MBR grub2-mbr cyl-align-off GPT
  Media summary: 1 session, 1431622 data blocks, 2796m data, 350g free
  Volume id : 'Ubuntu 20.10 amd64'
  System area options: 0x00004a00
  System area summary: MBR grub2-mbr cyl-align-off GPT
  ISO image size/512 : 5726488
  Partition offset : 16
  MBR heads per cyl : 0
  MBR secs per head : 0
  MBR partition table: N Status Type Start Blocks
  MBR partition : 1 0x80 0xcd 64 5715808
  MBR partition : 2 0x00 0xef 5715872 9952
  GPT : N Info
  GPT disk GUID : 7631942462da4c4797ccfb75ca94a436
  GPT entry array : 2 248 separated
  GPT lba range : 64 5726424 5726487
  GPT partition name : 1 490053004f004800790062007200690064003100
  GPT partname local : 1 ISOHybrid1
  GPT partition GUID : 1 7631942462da4c4797cdfb75ca94a436
  GPT type GUID : 1 a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags: 1 0x1000000000000001
  GPT start and size : 1 5715872 9952
  El Torito catalog : 678 1
  El Torito cat path : /boot.catalog
  El Torito images : N Pltf B Emul Ld_seg Hdpt Ldsiz LBA
  El Torito boot img : 1 BIOS y none 0x0000 0x00 4 679
  El Torito boot img : 2 UEFI y none 0x0000 0x00 9952 1428968
  El Torito img path : 1 /boot/grub/i386-pc/eltorito.img
  El Torito img opts...