Linux fails to read GPT(EFI) partition table created by both GNU parted and OSX diskutil

Bug #538100 reported by Eugene San on 2010-03-12
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
grub-installer (Ubuntu)
Medium
Colin Watson

Bug Description

Linux (linux-image-2.6.32-16-generic AMD64 2.6.32-16.25) fails to read GPT(EFI) partition table created by both GNU parted and OSX diskutil.

Test case:
1. dd if=/dev/zero of=/dev/sda count=128
2. parted:
    a. mklabel gpt
    b. mkpart primary fat 0 50
    c. quit
3. partprobe /dev/sda
4. dmesg | tail -n1

Kernel reports:
[ 5185.710614] sda: unknown partition table

Kernel config (grep -i part /boot/config-2.6.32-16-generic | grep -i efi):
CONFIG_EFI_PARTITION=y

Comments:
Ubiquity's partition manager working well with GPT partitions.
Ubiquity fails later during actual creation and formating of partitions: fails to access /dev/sdx devices.

tags: added: kernel-series-unknown
Jeremy Foshee (jeremyfoshee) wrote :

Eugene,
      Please run
apport-collect -p linux 538100
form the teerminal so that we can gather and post logs to this bug.

Thanks!

~JFo

Changed in linux (Ubuntu):
status: New → Incomplete
Eugene San (eugenesan) on 2010-03-12
tags: added: apport-collected
Eugene San (eugenesan) wrote :

I must mention that I am using Lucid daily AMD64 ISO for testing and looks like apport is currently pretty broken.
What information may be helpful?

Meanwhile I'll attach kern.log.

Eugene San (eugenesan) wrote :
Eugene San (eugenesan) on 2010-03-12
Changed in linux (Ubuntu):
status: Incomplete → New
Eugene San (eugenesan) on 2010-03-13
tags: removed: apport-collected
Eugene San (eugenesan) wrote :

Reconstructed on Karmic i386 (2.6.31-14-generic).

Jeremy Foshee (jeremyfoshee) wrote :

Eugene,
    It may be necessary to run 'sudo apt-get install apport' I had another user experience issue with apport today and running the install seems to have sorted him.

Thanks!

~JFo

Changed in linux (Ubuntu):
status: New → Incomplete
Eugene San (eugenesan) wrote :

Jeremy: What do you mean by install? In case you mean finish install and then report from fresh installation, then it's impossible since system may not be formatted...

Apport is still broken in latest daily ISO.
It crashes with python errors in parsing urls or something (under user and root).
Updating apport from repository fails also due to readonly root.

Anyways, I think issue might be related to:
1. http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commit;h=949c2699b38575324dd8522af6348b63df3e72dd
2. http://kernel.ubuntu.com/git?p=ubuntu/ubuntu-lucid.git;a=commit;h=f28d4f1e0a524ed8ec7039059ea6bc6a4b25a88b

Looks like second commit pretty destroys improvements first commit adds.

BTW: It might not looks like, but that bug is a show-stopper for many mac and 2TB+ storage users.

Eugene San (eugenesan) wrote :

I think I've localized the problem.
Looks Linux unable to deal with GPT pure partition tables (without synced MBR).
In past a patch (gptsync.patch) was added to parted to solve that issue. (Not sure ubiquity should enjoy from that fix also)
But it looks like parted refuses to sync GPT and MBR, even tried to force it by "PARTED_GPT_APPLE=1 parted".

Above also correct for OSX Graphical Disk Utility. Console version of OSX diskutil does create GPT with synced MBR.

Since the issue is not related to specific system, incomplete status should be removed.
Hope there is a solution floating in upstream...

tags: added: lucid
removed: kernel-series-unknown
Colin Watson (cjwatson) wrote :

It's not out of the question that this is actually a parted bug. I merged the Apple gptsync patch up to 2.2 more or less blindly, and I could have broken something.

I think you're analysing this incorrectly, though. What I'm seeing in the kernel is that you need a valid *protective* MBR, as specified in EFI. The gptsync patch in parted is to deal with the fact that Apple diverged from the GPT spec in requiring (approximately) a copy of the first four GPT partitions in MBR-speak rather than a dummy protective MBR whose only purpose is to say "hi, this is really GPT", so on Apple systems (only), parted has to act differently. A synced MBR is usually also a valid protective MBR as far as the kernel is concerned, but only if you have an EFI System Partition.

Therefore, I think the problem might well be that you haven't created an EFI System Partition, only a random FAT partition. Could you try this modification to your recipe?

  mklabel gpt
  mkpart primary fat 0 50
  set 1 boot on
  quit

Colin Watson (cjwatson) wrote :

See also bug 538536 for similar installer problems.

Eugene San (eugenesan) wrote :

Colin: Thanks for prompt respond.

Additional testing showed that problem is more severe then I assumed initially.
Testcase:
1) Zero HDD header.
2) Format HDD using OSX installer (3 partitions: fat32@EFI-Boot, hfs+@OSX, fat32@LINUX).
3) Reboot into OSX, partition table looks perfect.
3) Reboot into Daily Lucid. parted and fdisk shows expected output, correct Hybryd GPT+MBR
4) Use parted, ubiquity or partman to modify type and format of LINUX partiton.
5) Linux instantly reports inability to recognise partition table, partman and ubiquity fails.
    fdisk shows empty MBR. parted shows expected partitions layout but not sure about binary correctness.
6) Reboot and find that OSX cannot boot also!

Conclusion: parted and friends in Lucid makes GPT partition table not usable. System unbootable and data loss!

We will have to split the problem:
1) I suppose there is no reason why Linux shouldn't be able to work with bare GPT volumes. That part should be forwarded upstream for inspection.
2) parted and friends must be fixed ASAP. In their current state, they are source of destruction in GPT/EFI environments. We may consider merging that part with https://bugs.launchpad.net/bugs/538536

tags: added: debian-installer partman
Changed in linux (Ubuntu):
status: Incomplete → Confirmed
Eugene San (eugenesan) wrote :

I have tested GPT on non Mac system (VirtualBox).
1) Used gparted LiveCD to create GPT with 4 partitions. As result MBR contained protective MBR with single partition covering whole suface.
2) Booted in to latest Daily Lucid. Instructed installer to change type of 1 of 4 partitions and use 3 of 4 availbale.
3) Ubiquity finished successfuly.
4) Rebooted into fresh system.
5) Found that GPT is ok but MBR now contain 2 partitions:
    First one was previosly available protective
    Second was: Bootable, start=1, end=1, size=0, id=0 and type=Empty. What is that?

From all above I can conclude that issue is severe on Macs but non-Macs are not too safe either.
I think gpt-sync patch in parted must be reviewed ASAP, before people start testing Lucid betas on Macs...

Notice how the other bug I referenced is already targeted to beta-1 :-)

Eugene San (eugenesan) wrote :

Hope issue will be solved soon.
Since my Laptop is currently OS-less (at least for now) I am able to perform some testing if required.

Colin Watson (cjwatson) wrote :

Indeed, I'm working on it.

On "bare GPT volumes": the EFI specification requires that all GPT disks have a Protective MBR in the first LBA. I think Linux is entitled to check for this, if nothing else to avoid accidental misdetection of the partition table format.

Colin Watson (cjwatson) wrote :

I'd appreciate logs from the non-Mac case (specifically /var/log/installer/syslog and /var/log/installer/partman). I haven't been able to reproduce this behaviour in an isolated test of parted, and I'd like to see the sequence of operations that the installer is performing.

Eugene San (eugenesan) wrote :

Reconstructed the problem (with parted 2.2-1ubuntu2):
1. Clean GPT with 3 partitions (Gparted)
2. Start ubiquity (Lucid Daily)
    * Change type of 3rd partition
    * Format 2st and 3rd partitions (ext4 and swap)
    * Finish install
3. After reboot found partition table in state as below:
==========================================
# fdisk -l
WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.
Disk /dev/sda: 8589 MB, 8589934592 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical / optimal IO): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot Start End Blocks Id System
/dev/sda1 1 1045 8388607+ ee GPT
/dev/sda2 * 1 1 0 0 Empty
Partition 2 does not end on cylinder boundary.
==========================================
# parted -l
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 8590MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number Start End Size File system Name Flags
 1 17.4kB 206MB 206MB fat32 boot
 2 206MB 4401MB 4195MB ext4
 3 4401MB 8587MB 4187MB linux-swap(v1)
===========================================

Logs are attached.

BTW: Above didn't happen when disabled installation of grub into MBR, so it might be grub related issue...

Eugene San (eugenesan) wrote :
Colin Watson (cjwatson) wrote :

That's a grub-installer bug. Fixed and uploaded, pending release team approval.

Is there still a bug to handle against linux here? I don't think there's been any clear identification of a kernel bug, aside from the question of GPT labels without protective MBRs which I think I addressed in a previous comment.

Changed in grub-installer (Ubuntu):
assignee: nobody → Colin Watson (cjwatson)
importance: Undecided → Medium
status: New → Fix Committed
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package grub-installer - 1.49ubuntu7

---------------
grub-installer (1.49ubuntu7) lucid; urgency=low

  * Fix GPT detection to account for changed fdisk output (LP: #538100).
 -- Colin Watson <email address hidden> Mon, 15 Mar 2010 10:00:46 +0000

Changed in grub-installer (Ubuntu):
status: Fix Committed → Fix Released
Eugene San (eugenesan) wrote :

I assume updated parted and grub-installer are still expected to handle Mac partitioning demands, right?
After they will land in Daily Lucid, I'll test them on Mac and report.

About Linux, since Linux supports only Hibryd GPT, we would expect every partitionnig tool to prepare such, right?
Is it correct from EFI perspective?

Colin Watson (cjwatson) wrote :

On Mon, Mar 15, 2010 at 12:15:45PM -0000, Eugene San wrote:
> I assume updated parted and grub-installer are still expected to
> handle Mac partitioning demands, right?

Yes, should do.

> After they will land in Daily Lucid, I'll test them on Mac and report.

Thanks.

> About Linux, since Linux supports only Hibryd GPT, we would expect every partitionnig tool to prepare such, right?
> Is it correct from EFI perspective?

The EFI specification (which is also the specification for GPT) does not
define the term "Hybrid GPT". It specifies that all GPT disks have a
protective MBR. I don't think that makes them "hybrid" in any
significant sense - it's simply a part of the GPT specification.

Is there some more authoritative reference that you're thinking of here?

Eugene San (eugenesan) wrote :

I can't think of any authoritative contact.
Anyway, If existence of MBR (at least protective MBR) is vital part of GPT/EFI specification then it's ok for Linux to demand it.
Linux could just ignore absence of MBR in GPT, but we can't treat its behavior as bug.

Eugene San (eugenesan) wrote :

Results of testing on Mac:
  * parted do not create MBR when empty GPT is created.
  * ubiquity:
     a) When starting from empty GPT and creating partitions on GPT it creates synced MBR, but Linux then reads MBR instead of GPT.
     b) Created by ubiquity partitions created without alignment. Only last partition created with some spare space at end of device, while OSX Disk utility does insert holes and parted refuses to create without holes.
     c) When editing attributes of existing partitions it behaves normally, but only if partitions layout was created previously by ubuquty and synced to MBR using parted. In that case ubiquity do not destroy MBR.
     d) When starting with correct GPT (prepared by gparted or OSX Disk Utility), ubiquity converts holes in between first 3 partitions, to partitions in MBR.

Unfortunately ubiquity crashed few times and it's partitioning behavior was too hard to "catch", so there were no useful/sequential logs to provide.

In additiion I would like get help with definition of bugs, for two issues I have found while testing:
  * ubiquity do not provide ability to create new GPT, only MBR.
  * ubiquity do not provide ability to use grub-efi when installing to devices with GPT.

no longer affects: linux (Ubuntu)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers