Overlapping partitions created on Apple GPT

[Dave Vasilevsky]

Binary package hint: parted

Release: Natty daily 2011.04.10. Bug also occurs in Maverick and Lucid.
Package: parted 2.3-5ubuntu4

Parted in Ubuntu is patched to sync the hybrid-MBR with the GPT partition table on Apple systems. Under certain circumstances, a broken hybrid-MBR will be produced with multiple overlapping partitions of partition-ID 0xEE.

The effect can be reproduced, and the cause diagnosed, see TEST CASE and DIAGNOSIS below.

INCIDENCE AND EFFECTS:

A typical case where this bug may manifest is a dual- or triple-booting Mac. First the user installs Mac OS X, partition table looks like this:

1. EFI System Partition
2. Mac OS X (HFS+)

Next Ubuntu is installed alongside, partition table is now:

1. EFI System Partition
2. Mac OS X (HFS+)
3. Grub BIOS Boot Partition
4. Linux root (ext4)
5. Linux swap

The resulting hybrid MBR is:

1. EFI protective partition (0xEE)
2. Mac OS X (0xAF)
3. Grub BIOS Boot Partition (0xEE, overlaps partitions 1 & 2)
4. Linux root (0x83)

There are instances in the forums of such situations, eg:
http://ubuntuforums.org/showthread.php?t=1556993
http://ubuntuforums.org/showthread.php?t=1684602

The author of gdisk has also seen this bug: http://www.rodsbooks.com/gdisk/hybrid.html
'I'm seeing a lot of problem reports from Mac users who attempt to dual-boot with Linux because one of the utilities involved creates a buggy hybrid MBR with an extra type-0xEE partition that overlaps others.'

I will attach the output of 'parted print' and 'fdisk -l' for a triple-boot system where I encountered this bug.

Both Ubuntu and Mac OS X will continue to boot in my experience, as will Windows if present. However, disk utilities may become very confused, and may cause damage attempting to fix the broken partition table. Also, Mac OS X will believe it is not on a GPT-formatted disk, and will not be able to install firmware upgrades.

TEST CASE:

1. Add a new disk to an Ubuntu system or VM. I'll assume this is /dev/sda .

2. Create a GPT partition table with an EFI system partition:
 a. Run 'parted /dev/sda' as root
 b. In parted, create the table and partition:

  (parted) mktable gpt
  (parted) mkpart EFI fat32 1MiB 201MiB
  (parted) set 1 boot on
  (parted) quit

 c. Initialize the EFI system partition so parted will recognize it:
  # mkfs.vfat -F 32 /dev/sda1

3. Now we'll add some partitions, and see the bug.
 a. Run parted as root such that it will emulate running on an Apple
   system: 'PARTED_GPT_APPLE=1 parted /dev/sda'
 b. In parted, create a BIOS Boot Partition for grub2. We'll also create
  some extra partitions to make the problem easier to notice.

  (parted) mkpart HFS hfs+ 201MiB 500MiB
  (parted) mkpart grub ext2 500MiB 502MiB
  (parted) set 3 bios_grub on
  (parted) mkpart linux-root ext4 502MiB 700MiB
  (parted) quit

4. Examine the MBR with 'fdisk -l /dev/sda'. The results:

   Device Boot Start End Blocks Id System
/dev/sda1 1 26 205823+ ee GPT
/dev/sda2 * 26 64 306176 0 Empty
/dev/sda3 1 64 514047+ ee GPT
/dev/sda4 64 90 202752 83 Linux

Notice that the BIOS Boot Partition (sda3) overlaps the first two partitions. The expected result would be something like:

   Device Boot Start End Blocks Id System
/dev/sda1 1 26 205823+ ee GPT
/dev/sda2 * 26 64 306176 0 Empty
/dev/sda3 64 64 2048 da Non-FS data
/dev/sda4 64 90 202752 83 Linux

Or perhaps even:

   Device Boot Start End Blocks Id System
/dev/sda1 1 26 205823+ ee GPT
/dev/sda2 * 26 64 306176 0 Empty
/dev/sda3 64 90 202752 83 Linux

(Device name /dev/sda3 should be sda4, bug in fdisk?)

DIAGNOSIS:

The proximate cause of this bug is line 317 in gptsync.patch, which sets the StartingLBA = 1 for any partition of type 0xEE. This should be changed to only apply to the FIRST partition.

http://bazaar.launchpad.net/~ubuntu-branches/ubuntu/natty/parted/natty/view/head:/debian/patches/gptsync.patch

[Dave Vasilevsky (djvasi)]

Formatting of fdisk output is pretty borked, is there a way to fix that?

In any case, here are the pre- and post-install partition tables when installing Ubuntu on my triple-boot system:

PRE GPT:
Number Start End Size File system Name Flags
        17.4kB 20.5kB 3072B Free Space
 1 20.5kB 210MB 210MB fat32 EFI System Partition boot
 2 210MB 160GB 160GB hfs+ Macintosh HD
        160GB 160GB 135MB Free Space
 3 160GB 320GB 160GB ntfs Windows
        320GB 320GB 335kB Free Space

PRE MBR
   Device Boot Start End #cyls #blocks Id System
/dev/sdb1 0+ 25- 26- 204819+ ee GPT
/dev/sdb2 25+ 19481- 19457- 156282164 af HFS / HFS+
/dev/sdb3 * 19498+ 38913- 19416- 155952128 7 HPFS/NTFS
/dev/sdb4 0 - 0 0 0 Empty

POST GPT
Number Start End Size File system Name Flags
        17.4kB 20.5kB 3072B Free Space
 1 20.5kB 210MB 210MB fat32 EFI System Partition boot
 2 210MB 160GB 160GB hfs+ Macintosh HD
        160GB 160GB 135MB Free Space
 3 160GB 239GB 78.5GB ntfs Windows
        239GB 239GB 848kB Free Space
 4 239GB 239GB 1049kB bios_grub
 5 239GB 317GB 77.9GB ext4
 6 317GB 320GB 3352MB linux-swap(v1)
        320GB 320GB 335kB Free Space

POST MBR
   Device Boot Start End #cyls #blocks Id System
/dev/sdb1 0+ 25- 26- 204819+ ee GPT
/dev/sdb2 * 25+ 19481- 19457- 156282164 af HFS / HFS+
/dev/sdb3 19498+ 29040- 9543- 76650692 7 HPFS/NTFS
/dev/sdb4 0+ 29040- 29041- 233271295+ ee GPT

Notice that sdb4 overlaps sdb1 in the post-install hybrid-MBR.

[Dave Vasilevsky (djvasi)]

Aside from the proximate bug in gptsync.patch, there are two reasons I think this shouldn't have become an issue in the first place. I'd appreciate any thoughts.

1. Not all partition types should go in the hybrid MBR

The hybrid MBR exists for just two reasons:
i. So non-GPT-aware OSes can access GPT partitions
ii. So BIOSes/bootloaders that use the MBR exclusively can jump to the Partition Boot Record of GPT partitions, for booting.

Neither of these uses are relevant to the grub BIOS Boot Partition. It's just raw code/data, there's no reason for a non-GPT-aware OS to touch it. And no foreign bootloader should ever jump directly to it, only grub should do so, and grub doesn't need the MBR to find it. Other partition types, such as Linux Swap, also have no use in a hybrid-MBR.

Parted's gptsync should therefore omit BBP and Linux Swap partitions from the hybrid-MBR.

2. Ubuntu's installer should use an EFI System Partition on GPT

Although it's not mandatory, many OSes and tools expect a GPT partition table to begin with an EFI System Partition. The elilo bootloader uses it for booting Linux. Mac OS X's disk utility and installer create an ESP on every GPT disk they format, and the OS can't do firmware updates without one. The rEFIt bootloader is often installed in the ESP. I'm not sure what Windows does, but MS's docs recommend that an ESP comes first in the GPT and MBR.

Ubuntu's installer currently does not create such an ESP on a clean GPT disk. Instead, to satisfy the Linux kernel's GPT checks, it assigns to the BIOS Boot Partition the MBR ID for a protective EFI partition, 0xEE. (See https://bugs.launchpad.net/ubuntu/+source/partman-auto/+bug/538536 )

I don't think this is the right solution. If later another MBR/GPT-syncing tool (rEFIt, gdisk) is used, it will not give the BBP the MBR ID 0xEE, and booting Linux will fail. Other OSes may not install without an ESP, or may be otherwise unhappy.

Partman-auto should therefore put an ESP at the start of a GPT disk when possible. An example partition layout looks entirely reasonable:

GPT
1. EFI System Partition
2. BIOS Boot Partition
3. Linux root (ext4)
4. Linux swap

Hybrid MBR
1. EFI protective partition
2. Linux root (ext4)

[Colin Watson (cjwatson)]

Thanks for your detailed report. I'll look into fixing this as soon as I can.

Regarding item 1 (gptsync behaviour for irrelevant partitions), while in theory I agree, I wonder if it is wise to change that at this point. Changing that could mean that partition numbers change if people run a libparted-based program and just change one unrelated detail, and it strikes me that that is not unlikely to cause problems. I am leaning towards leaving this alone.

[Dave Vasilevsky (djvasi)]

cjwatson, thanks so much for taking a look at this.

I hear your concern about changing whether irrelevant partitions are placed in the hybrid MBR. Note however that the only partition numbering that would change would be in the MBR. In a GPT + hybrid-MBR configuration, linux takes its partition numbers exclusively from the GPT, so linux device numbers won't change at all. Are you aware of any utilities or OSes that care particularly about the index of partitions within a hybrid-MBR?

If there are tools that care, another possibility is to simply zero out the entries corresponding to irrelevant partitions. They'll then be ignored, but the index of other partitons will remain the same.

[Phillip Susi (psusi)]

At the risk of going off topic, when installing on an EFI mac, shouldn't grub-efi be installed to the EFI system partition, and there should not be a grub bios boot partition at all?

[Dave Vasilevsky (djvasi)]

I was just doing some testing and realized that there's a bigger problem: If a disk has no EFI system partition, and no grub_bios partition, then the hybrid-MBR we create doesn't contain an 0xEE partition at all, and Linux will not recognize the disk as GPT! We should definitely special-case this and refuse to create a hybrid MBR.

[Dave Vasilevsky (djvasi)]

Here's a patch to libparted/labels/gpt.c that's a first stab at a solution. The changes:

1. Only the first partition may start at sector 1, don't modify later partitions that way. This avoids overlap.

2. If no protective partition at all is found, place at the start of the MBR a tiny fake protective partition, covering the space from sector 1 to the start of the first real partition. Partitions that follow are moved up by one, so that the hybrid MBR only includes three real partitions. (This approach is used by gpt-fdisk, see: http://www.rodsbooks.com/gdisk/hybrid.html )

This means if your GPT looks like this:

1. HFS: 1MB to 1GB
2. NTFS: 1GB to 2GB
3. ext3: 2GB to 3GB
4. ext4: 3GB to 4GB

The hybrid MBR will look like:

1. Protective: 512b to 1MB
2. HFS: 1MB to 1GB
3. NTFS: 1GB to 2GB
4. ext3: 2GB to 3GB

Critique:

* Creation of a fake protective partition causes the partition numbering in the MBR to change. However, since parted currently produces partition tables that can't be read by Linux, this is almost certainly worth it.

* It might be best to create a fake initial protective partition in all cases where the first partition is not of type 0xEE. Thoughts?

* No attempt is made to leave certain partition types out of the hybrid MBR, for now (see issue #1 in my previous comment).

[Colin Watson (cjwatson)]

Thanks! I'll look at this in detail before I finish today.

Re your question about a fake initial protective partition, I think we can probably leave things as they are for natty. Fixing bug 746313 should reduce the incidence of this, and TBH I think that GRUB's behaviour of requiring the first partition to be the protective one should be changed anyway.

[Colin Watson (cjwatson)]

Regarding what cares about partition indices in the hybrid MBR: well, I would expect that'd be legacy systems that require the MBR in the first place, so indeed naturally not Linux. However, I guess in most situations where you're trying to use such systems on Macs there'll be an EFI System Partition anyway, so any problems caused by this should be pretty limited.

[Colin Watson (cjwatson)]

I'm going to go ahead with your patch for now. It definitely looks like an improvement. Thanks again!

[Dave Vasilevsky (djvasi)]

Thanks again cjwatson! I love when bug reports get such quick turnaround :)

[Rod Smith (rodsmith)]

I'm the author of GPT fdisk (gdisk, sgdisk). Some comments:

* There's no such thing as a "fake protective partition." The 0xEE partition in the MBR does not correspond to any real partition in the GPT. The purpose of the 0xEE partition is to keep GPT-unaware utilities from messing with the disk. It ordinarily covers the space from sector 1 to the end of the disk, but on a hybrid disk this is reduced to the space from sector 1 to one less than the first sector of the first (in disk space terms) GPT partition included in the hybrid MBR. Note in particular that the 0xEE partition does NOT correspond to the EFI System Partition (ESP), although at first glance it might look like it does. (The belief that these two partitions are equivalent can cause problems if a user attempts to access the 0xEE partition from Windows. If writes are attempted, the result can be damage to the GPT data structures!)

* IMHO, routinely creating a hybrid MBR is a Bad Idea (note the i-caps). See my Web page on hybrid MBRs (http://www.rodsbooks.com/gdisk/hybrid.html) for an extended discussion of this, but in brief, they're ugly and dangerous -- it's too easy for either an MBR partitioning tool or a GPT partitioning tool to get the two to be out of sync in one way or another, which can have disastrous results. Unfortunately, Apple chose to rely on hybrid MBRs to get Windows booting on Macs, so we're left dealing with their mess. In an ideal world, Linux could boot on Macs using EFI mode, obviating the need for a hybrid MBR on Linux/MacOS dual-boot systems, but this remains a bit tricky. As to rules for when to create a hybrid MBR automatically, that's even trickier, but I'd start with doing it ONLY on Macs or on BIOS-based systems that already have one. On UEFI-based PCs, a hybrid MBR should never be necessary (since Windows boots fine in UEFI mode from GPT on such systems). On a BIOS-based PC, a hybrid MBR might be necessary to get Windows booting (as in a Hackintosh configuration), but Linux doesn't need it, so there's no point in creating one if it doesn't already exist. I haven't looked at the code in question so I don't know how hard it would be to add such rules to it, but blindly creating hybrid MBRs when they aren't necessary is a recipe for trouble.

* I wouldn't worry about keeping GPT and MBR partition numbers in sync. I don't know of any OS or utility that cares about this detail, and as I wrote earlier, the MBR's 0xEE partition isn't in sync with anything on the GPT side to begin with.

* I agree with Dave that there's no point in including a BIOS Boot Partition or Linux swap partition in a hybrid MBR. I'd go further: there's little point in including ANY Linux or OS X partition in a hybrid MBR; neither Linux nor OS X uses the hybrid MBR at all, and GRUB reads the GPT data just fine. I know of only two reasons to include such a partition: To kick Apple's EFI into BIOS compatibility mode to enable booting Linux in that way; and to enable Windows to access a Linux or OS X partition when appropriate drivers are installed. Given all the other variables at work, IMHO it's best to err on the side of not including these partitions in the hybrid MBR (and thus conve...

[Dave Vasilevsky (djvasi)]

Rod, the gptsync code in parted is (and has been) only enabled on Intel Macs. Other machines will create regular old protective MBRs.

[Launchpad Janitor (janitor)]

This bug was fixed in the package parted - 2.3-5ubuntu5

---------------
parted (2.3-5ubuntu5) natty; urgency=low

  [ Dave Vasilevsky ]
  * Improve built-in gptsync implementation (LP: #757201):
    - Only the first partition may start at sector 1; don't modify later
      partitions that way. This avoids overlap.
    - If no protective partition at all is found, place at the start of the
      MBR a tiny fake protective partition, covering the space from sector 1
      to the start of the first real partition. Partitions that follow are
      moved up by one, so that the hybrid MBR only includes three real
      partitions.
 -- Colin Watson <email address hidden> Fri, 15 Apr 2011 17:29:54 +0100

[Erick Brunzell (lbsolost)]

Possible regression reported at bug 765204.

[demilord (demilord)]

Nope was a hardware issue, a BIOS setting, I think this one can be closed now

[JC Hulce (soaringsky)]

This bug affects Ubuntu 10.10, Maverick Meerkat. Maverick has reached end-of-life and is no longer supported, so I am closing the bugtask for Maverick. Please upgrade to a newer version of Ubuntu.
More information here: https://lists.ubuntu.com/archives/ubuntu-announce/2012-April/000158.html

[Rolf Leggewie (r0lf)]

lucid has seen the end of its life and is no longer receiving any updates. Marking the lucid task for this ticket as "Won't Fix".