hybrid images do not include a valid GPT

[Steve Langasek]

The insyde UEFI firmware implementation does not see the GPT on the hybrid images generated on nusakan (e.g., the 12.10 beta2 image). This appears to be consistent with what both gdisk and parted show:

$ gdisk -l ~/devel/iso/ubuntu-12.10-beta2-desktop-amd64.iso
GPT fdisk (gdisk) version 0.8.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present

***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format.
***************************************************************

Disk /home/vorlon/devel/iso/ubuntu-12.10-beta2-desktop-amd64.iso: 1548288 sectors, 756.0 MiB
Logical sector size: 512 bytes
Disk identifier (GUID): 1ADDF8C9-5CE1-41A4-9124-2CC80EED991F
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1548254
Partitions will be aligned on 2048-sector boundaries
Total free space is 1548221 sectors (756.0 MiB)

Number Start (sector) End (sector) Size Code Name
$

Removing the -partition_offset 16 option from the xorriso commandline is sufficient to generate an image with a recognizable partition table; but I imagine this causes other problems for hybrid systems.

[Steve Langasek (vorlon)]

The same behavior is reproducible in OVMF.

[Steve Langasek (vorlon)]

diff of the first 4k of the images, showing the differences between the broken one (with -partition_offset 16) and the working one (without):

@@ -21,17 +21,17 @@
 0000500 07 cd 10 3c 0a 75 f1 cd 18 f4 eb fd 00 00 00 00
 0000520 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 *
-0000660 10 76 17 00 00 00 00 00 b5 72 cf 26 00 00 80 02
-0000700 01 00 00 3f a0 f3 40 00 00 00 c0 9f 17 00 00 00
-0000720 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
-*
+0000660 98 a1 16 00 00 00 00 00 7c 72 04 3b 00 00 80 00
+0000700 01 00 00 3f a0 f5 00 00 00 00 00 b0 17 00 00 fe
+0000720 ff ff ef fe ff ff ec 99 17 00 40 11 00 00 00 00
+0000740 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 0000760 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa
 0001000 45 46 49 20 50 41 52 54 00 00 01 00 5c 00 00 00
-0001020 13 19 f0 f0 00 00 00 00 01 00 00 00 00 00 00 00
-0001040 ff 9f 17 00 00 00 00 00 40 00 00 00 00 00 00 00
-0001060 ca 9f 17 00 00 00 00 00 c7 68 7d 1f 03 74 68 40
-0001100 92 63 94 78 cb 8f 38 ff 0c 00 00 00 00 00 00 00
-0001120 d0 00 00 00 80 00 00 00 97 32 fd d7 00 00 00 00
+0001020 7c 5c 66 d4 00 00 00 00 01 00 00 00 00 00 00 00
+0001040 ff af 17 00 00 00 00 00 40 00 00 00 00 00 00 00
+0001060 ca af 17 00 00 00 00 00 95 6a a5 6e 23 51 6a 40
+0001100 92 63 c6 61 97 82 b7 e9 0c 00 00 00 00 00 00 00
+0001120 d0 00 00 00 80 00 00 00 c8 eb 58 a4 00 00 00 00
 0001140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
 *
 0004000 50 4d 00 00 00 00 00 02 00 00 00 01 00 00 00 02

[Thomas Schmitt (scdbackup)]

Hi,

being developer of xorriso i wonder how leaving out
-partition offset 16 can cause a second MBR partition

+0000700 00 fe
+0000720 ff ff ef fe ff ff ec 99 17 00 40 11 00 00

It appears to be an EFI partition of ~ 2.2 MB.

Is the ISO image subject to further manipulations after
it was produced by xorriso ?

Where can i get the exact xorriso command and a resulting
ISO 9660 image in order to make own experiments ?
(I do not have an EFI system but i can at least inspect the
 partition tables.)

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

i forgot to comment on this statement:

> Removing the -partition_offset 16 [...]
> I imagine this causes other problems for hybrid systems.

The feature is mostly for beautification of MBR in order
to make it similar to conventional HD MBRs. It shall offer
an unclaimed "embedded area" where boot loaders can install
themselves and point to added partitions on the USB stick.

So far i have reports of one boot success where -partition_offset 0
fails, and one problem where -partition_offset 16 prevents mounting
on MacOS.

http://libburnia-project.org/wiki/PartitionOffset

Have a nice day :)

Thomas

[Steve Langasek (vorlon)]

Hi Thomas,

Thanks for the interest!

Here's the full command line being used for our image build currently:

xorriso -as mkisofs -r -checksum_algorithm_iso md5,sha1 -V Ubuntu\ 12.10\ amd64 -o /srv/cdimage.ubuntu.com/scratch/ubuntu/daily-live/debian-cd/amd64/quantal-desktop-amd64.raw -isohybrid-mbr syslinux/usr/lib/syslinux/isohdpfx.bin -partition_offset 16 -cache-inodes -J -l -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -eltorito-alt-boot -e boot/grub/efi.img -no-emul-boot -isohybrid-gpt-basdat -isohybrid-apm-hfsplus boot1 CD1

So the goal here is to be able to boot this image as a CD or USB disk image, under either BIOS or UEFI (and including the Mac UEFI implementation). I was assuming the -partition_offset 16 was relevant for MBR booting, but perhaps I'm mistaken.

[Steve Langasek (vorlon)]

Oh, answering your other questions:

> Is the ISO image subject to further manipulations after
> it was produced by xorriso ?

No; this problem is reproducible using just the output of xorriso (version 1.2.4-0ubuntu1).

> Where can i get the exact xorriso command and a resulting
> ISO 9660 image in order to make own experiments ?
> (I do not have an EFI system but i can at least inspect the
> partition tables.)

The image is available at <http://releases.ubuntu.com/12.10/ubuntu-12.10-beta2-desktop-amd64.iso>. It probably also affects the images at <http://cdimage.ubuntu.com/daily-live/current/>; I haven't yet tested those to be sure, but that's where we'll be iterating any attempts at fixing.

As for having EFI systems, yes, I think as a first pass we want to be outputting something that gdisk/parted is happy with the GPT on. For testing the actual boot under UEFI, I highly recommend using qemu together with ovmf, an open UEFI firmware implementation derived from Tiano (the Intel ref. implementation).

[Thomas Schmitt (scdbackup)]

Hi,

> Thanks for the interest!

Purely selfish. :))
Possibly you found a bug.

I was not notified about your new comments, although i subscribed
to #1062737.
So if i fail to answer, ping me at e.g <email address hidden>.

> I was assuming the -partition_offset 16 was relevant for MBR booting,

ISOLINUX isohybrid does not depend on it. H. Peter Anvin proposed such
a feature, in order to give isohybrid a more conventional partition
table.

> So the goal here is to be able to boot this image as a CD or USB
> disk image, under either BIOS or UEFI (and including the Mac UEFI
> implementation).

I see. mjg's condensed multi-boot contraption. You are probably
the first users of its implementation inside libisofs.

Currently it looks as if libisofs (underneath xorriso) did not
mark the EFI boot image as MBR partition when partition offset
was 16. Well, that's not promised for -isohybrid-gpt-hfsplus but
obviously happens for partition offset 0.

The GPT partition headers only differ where CRCs and random GUID
are stored. The partition entries are not shown, regrettably.
They are announced to start at 512-byte block 12 = 0014000 octal.
(You only diffed 2 kB. Better would have been all 32 kB of the ISO System
Area. So one could see the Apple Partition Map, too)

I am downloading the amd64.iso now. Will report on findings ...

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

(Finally the notification mails arrived for your comments #5 and #6.)

http://releases.ubuntu.com/12.10/ubuntu-12.10-beta2-desktop-amd64.iso
currently has 1 MBR partition and 2 GPT partitions.

The MBR partition claims the range of the whole ISO image.

GPT partition 1 does about the same. (Minus backup GPT.)
GPT partition 2 claims 832 blocks inside the image.
Its start address is the address of file /boot/grub/efi.img .
This file has a size of 425984 = 832 * 512.
So GPT looks like it should.

The missing MBR partition entry seems to be the only
deviation from mjg's layout for now. Except the start of
MBR partition 1 at 512-block 64 rather than 0.

I am puzzled by the size of MBR partition 2 of the +image
in your diff. It claims to have 0x1140 = 4416 blocks.
I would have expected 832 as with GPT.
Can it be that efi.img is not the same in both images ?

I will examine the source code and make experiments tomorrow.

BTW:
I documented mjg's layout in
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/view/head:/doc/boot_sectors.txt
  "SYSLINUX isohybrid for MBR, UEFI and x86-Mac"
See also
  "Master Boot Record (MBR), for PC-BIOS x86 from (pseudo-) hard disk"
  "Apple Partition Map (APM), for more modern Mac"
  "GUID Partition Table (GPT), for EFI from (pseudo-) hard disk"

Your layout deviates from mjg's by not having an HFS+ image embedded
in the ISO and exposed by the Apple Partition Map.

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

some misc notes before i go to bed:

The fact that the GPT is not recognized by partition editors
is a consequence of Matthew Garrett's (mjg) layout.
  http://mjg59.dreamwidth.org/11285.html
According to wikipedia, GPT is in effect when only a "protective
MBR" exists. mjg's rich MBR does not qualify for that.

I wrote in comment #7 "-isohybrid-gpt-hfsplus" where it should
have been "-isohybrid-gpt-basdat".

The announcement of efi.img as HFS+ partition in APM is
probably useless. As far as i understood Matthew, the APM is
introduced to point to the third boot image which would be
a HFS+ image file. The other two APM partitions are for
user information purpose only.
You do not submit a HFS+ file by option -e. So you will
take at boot time no advantage from option -isohybrid-apm-hfsplus.
Needed would be something like
  -eltorito-alt-boot \
  -e my_hfsplus_image -no-emul-boot \
  -isohybrid-gpt-hfsplus -isohybrid-apm-hfsplus \
before
  boot1 CD1
This would then form the complete mjg layout.
(About the way to generate the HFS+ image file i only have this
 link
   http://git.fedorahosted.org/cgit/lorax.git/tree/src/sbin/mkefiboot?id=d546b0e0f474cbe041b7916dcc9fcf04b51afb2a;id2=HEAD
)

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

it is indeed a bug in xorriso.
The program parts for mjg's layout nicely prepare a MBR partition 2
as mirror of the GPT partition which marks the VFAT image file.
But then comes the code for partition offset 16, manipulates MBR
partition 1 and zeroizes the other three MBR partitions.

I will try to fix this now. (Have to examine the reason for
the zeroizing first. It is older than partition_offset and GPT
support. Simply omitting it smells like regression.)
If i succeed, i will prepare a GNU xorriso-1.2.5 development
snapshot tarball.

With the released xorriso-1.2.4 (resp. libisoburn-1.2.4) one
will have to omit -partition_offset 16 in order to get
-isohybrid-gpt-basdat into effect in MBR.

Be aware that your current boot test obviously did not use
GPT or APM partitions but the ones of MBR.

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

the bug, which actually was in libisofs, should now be fixed by
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/revision/1044
I would appreaciate a boot test.

If you have a source package of libisofs-1.2.4 installed, you may
change the only occurence of the call iso_offset_partition_start().
See:
  http://bazaar.launchpad.net/~libburnia-team/libisofs/scdbackup/revision/1044/libisofs/system_area.c

A GNU xorriso development snapshot is ready at
  http://www.gnu.org/software/xorriso/xorriso-1.2.5.tar.gz
This is the standalone version without dynamic libs which can be
used without installation:
  configure && make
Execute as:
  ./xorriso/xorriso

Have a nice day :)

Thomas

[Steve Langasek (vorlon)]

Thanks for your work on this, Thomas. My understanding is that for the moment we can just drop the -partition_offset flag without causing any regressions in functionality, so that's what I think we should do since that's the easiest and we're very close to release deadlines. I've committed this change to the build script repo for now.

We can revisit enabling this option after 12.10 when we pull in xorriso 1.2.5, or if any problems do turn up in testing with -partition_offset disabled (which seems unlikely).

[Steve Langasek (vorlon)]

I've built a test image now using the cherry-picked patch from 1.2.5, and confirmed that the -partition_offset option now happily coexists with -isohybrid-gpt-basdat in all of my testing (USB boot on real UEFI hardware; USB and CD boot in BIOS VM; USB boot in UEFI VM). So that seems to be a suitable fix.

However, as the resultant image still appears to not have a valid MBR partition table, there doesn't seem to be any advantage to adding this option back to our builds for 12.10. So I think we'll leave it as is with -partition_offset 16 disabled for now.

[Thomas Schmitt (scdbackup)]

Hi,

> I've built a test image now using the cherry-picked patch from 1.2.5,
> and confirmed that the -partition_offset option now happily coexists
> with -isohybrid-gpt-basdat

Thanks a lot for testing.

> However, as the resultant image still appears to not have a valid MBR
> partition table, there doesn't seem to be any advantage to adding this
> option back to our builds for 12.10. So I think we'll leave it as is
> with -partition_offset 16 disabled for now.

Do you rather mean "no valid GPT" ?
The MBR seems to be ok now. After all it made the difference
between boot and not-boot.

The GPT is there but it usually not recognized because the MBR
is not trivial enough (not a "protective MBR"). This should have
nothing to do with -partition_offset.
If it has, then this would be probably another bug.

I googled a bit and found:
  http://www.rodsbooks.com/gdisk/hybrid.html
The described "Hybrid MBR" would differ from mjg's mainly by the
type of the first MBR partition: 0xee rather than 0x00 im mjg's.

So does it help GPT visibility to set byte 450 of the ISO image
to the value 0xee ? (Byte count starting at 0)

Have a nice day :)

Thomas

[Steve Langasek (vorlon)]

On Mon, Oct 08, 2012 at 07:53:10PM -0000, Thomas Schmitt wrote:
> > However, as the resultant image still appears to not have a valid MBR
> > partition table, there doesn't seem to be any advantage to adding this
> > option back to our builds for 12.10. So I think we'll leave it as is
> > with -partition_offset 16 disabled for now.

> Do you rather mean "no valid GPT" ?

That's not what I mean, but it's possible that I'm mistaken. UEFI looks for
an EFI system partition, which is supposed to be defined as a GPT partition
of type EF00 containing a FAT filesystem. Since this image now boots under
UEFI, UEFI is satisfied that it's found a system partition - which I was
assuming meant a valid GPT. However, I now see that an MBR partition of
type 'EF' will be up-converted to a GPT partition of type 'EF00', which I
guess is what is happening here.

> The MBR seems to be ok now. After all it made the difference
> between boot and not-boot.

fdisk remains unhappy with the image, but this may be a separate issue:

Warning: /home/vorlon/devel/iso/quantal-desktop-amd64.iso contains GPT
signatures, indicating that it has a GPT table. However, it does not have a
valid fake msdos partition table, as it should. Perhaps it was corrupted --
possibly by a program that doesn't understand GPT partition tables. Or perhaps
you deleted the GPT table, and are now using an msdos partition table. Is
this a GPT partition table?
   y Yes
   n No

gdisk in turn reports that the image is MBR-only.

> The GPT is there but it usually not recognized because the MBR
> is not trivial enough (not a "protective MBR"). This should have
> nothing to do with -partition_offset.
> If it has, then this would be probably another bug.

> I googled a bit and found:
> http://www.rodsbooks.com/gdisk/hybrid.html
> The described "Hybrid MBR" would differ from mjg's mainly by the
> type of the first MBR partition: 0xee rather than 0x00 im mjg's.

> So does it help GPT visibility to set byte 450 of the ISO image
> to the value 0xee ? (Byte count starting at 0)

That doesn't seem to make any difference in whether fdisk/gdisk are happy
with it, at least.

[Thomas Schmitt (scdbackup)]

Hi,

> Since this image now boots under
> UEFI, UEFI is satisfied that it's found a system partition - which I was
> assuming meant a valid GPT. However, I now see that an MBR partition of
> type 'EF' will be up-converted to a GPT partition of type 'EF00', which I
> guess is what is happening here.

At leat the existence of that MBR partition is the difference
between libisofs with and witout the bug fix. GPT did not change
by this.

> fdisk remains unhappy with the image, but this may be a separate issue:
> Warning: /home/vorlon/devel/iso/quantal-desktop-amd64.iso contains GPT
> signatures, indicating that it has a GPT table. However, it does not have a
> valid fake msdos partition table, as it should.

Yes. That is most likely the reason why you need the MBR partition.
It is a feature of mjg's layout.
http://mjg59.dreamwidth.org/11285.html states
"it turns out that there are some systems that refuse to BIOS-boot
 off GPT media. So we need an MBR partition map. The first entry
 covers the entire disk and is of type 0. This is important, because
 some EFI implementations are strict about MBR parsing - if there's
 an MBR with overlapping partitions, the entire MBR will be ignored.
"
"Why have a GPT at all? The EFI spec says that machines should boot
 fine from an MBR partition. Sadly, not all seem to.
"

> > So does it help GPT visibility to set byte 450 of the ISO image
> > to the value 0xee ? (Byte count starting at 0)

> That doesn't seem to make any difference in whether fdisk/gdisk are happy
> with it, at least.

Pity. Did you check that the manipulation really changed
only byte 450(dec) from 0x00 to 0xee ?
I had hoped that at least gdisk would accept this. The description
hybrid.html obviously stems from gdisk documentation.

I assume you do not see a valid GPT without -partition_offset
either.
If you see a GPT, then please correct me.
In that case, the partition start at 64 blocks could indeed
be the culprit that deceives gdisk.

The normal way to make GPT visible would be to reduce the
MBR partition table to form a "protective MBR".
To achieve a protective MBR you would have to zeroize the bytes
462(dec) to 509 in the ISO image (made without -partition_offset).
Then you would have to set byte 450(dec) to 0xee (again).

This would possibly not boot on all systems but should at
least make the GPT recognizable.

Have a nice day :)

Thomas

[Thomas Schmitt (scdbackup)]

Hi,

i closed this bug report because the libisofs fix is released since
nearly four years.

Summary of remaining issue:

The isohybrid layout as defined for ISOLINUX bootable ISO 9660 images
regularly works by the MBR partition of type 0xef, not by the GPT partition
which carries the type UUID of EFI System Partition.
This is well compliant to UEFI specs. E.g. UEFI 2.4, 5.2.1 "MBR".
Under these specs, the GPT and its backup are just opaque bytes of no
special meaning to UEFI, because the MBR is not "Protective".

GRUB2 program grub-mkrescue on the other hand lets xorriso create a
Protective MBR and a valid GPT with a partition entry which marks the
EFI System Partition.

Have a nice day :)

Thomas