Incorrect GPT length should be handled better

[Jakey_TheSnake]

The GPT LastUsableLba claims the disk is longer than it actually is. Parted should handle this error more gracefully and allow for repairing it. gdisk seems to let the error go undetected, possibly because it is using the size in the protective MBR rather than LastUsableLba.

[Curtis Gedak (gedakc)]

Does the crash occur if you run "sudo gparted" from a terminal?

If so can you post the text output?

[Phillip Susi (psusi)]

Can you please run the following command and attach the resulting file, replacing sda with the correct drive designation:

dd if=/dev/sda bs=512 count=64 | gzip -c > partition-table.gz

[Launchpad Janitor (janitor)]

[Expired for gparted (Ubuntu) because there has been no activity for 60 days.]

[Ahmad Moawya (ahmadmoawya)]

Here are my findings thus far:

1- Ubuntu does not detect any partitions during boot. (don't see them in the file manager)

2- Checking the system log, I get "unknown partition map" or something similar to this.

3- Gparted (and parted-server when I run the installer) crash.
Runnung parted thru terminal, I get "Assertion (last_usable <= disk->dev->length) at ../../../libparted/labels/gpt.c:718 in function _parse_header() failed."

I verified my disk using GPT fdisk (gdisk) and verification was successful (found valid GPT and protective MBR)
Mac OS X disk utility verification finds no problems whatsoever.
Interestingly, I tried an Ubuntu 10.04 live CD I had and it detected the partition, Gparted works fine, partition are listed in nautilus, and I can mount them.

I suspect this issue is either related to parted binary or the newer linux kernel.

Feel free if you have any questions.

[Ahmad Moawya (ahmadmoawya)]

Here is the GPT data backed up by GPT fdisk (gdisk).

[Phillip Susi (psusi)]

Can you add the output of fdisk -lu?

That assertion failure indicates that the partition table claims that the disk is larger than it actually is.

[Ahmad Moawya (ahmadmoawya)]

Here is the output of fdisk -lu

WARNING: GPT (GUID Partition Table) detected on '/dev/sda'! The util fdisk doesn't support GPT. Use GNU Parted.

Disk /dev/sda: 160.0 GB, 160040803840 bytes
256 heads, 63 sectors/track, 19381 cylinders, total 312579695 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot Start End Blocks Id System
/dev/sda1 1 312579694 156289847 ee GPT

Disk /dev/sdb: 1028 MB, 1028521984 bytes
255 heads, 63 sectors/track, 125 cylinders, total 2008832 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot Start End Blocks Id System
/dev/sdb1 * 63 2008817 1004377+ b W95 FAT32

[Ahmad Moawya (ahmadmoawya)]

This message in system log during boot has particularly caught my attention:

ubuntu kernel: [ 4.322503] sda: unknown partition table

Could the problem be kernel related?

[Phillip Susi (psusi)]

Your partition table is indeed corrupt. It claims the disk should be 312581808 sectors long, not 312579695. It seems that gdisk fails to detect and diagnose this error. Parted should handle this error more gracefully than it does. It seems that current upstream parted just refuses to recognize the partition table rather than crashing with an assertion failure. Patching parted to be able to correct such an error has been on my todo list.

[Rod Smith (rodsmith)]

I'm the author of gdisk; psusi e-mailed me suggesting I look at this bug report.

I believe the problem stems from an inconsistency in motherboard-based RAID configuration or kernel drivers. Note that fdisk is reporting a disk size of 312579695 sectors. This is an odd number, both in the mathematical sense and in the sense that most (in fact, all, to the best of my knowledge) hard disks today have an even number of sectors. The only cases when I've seen modern disks with an odd number of sectors are when those disks are being accessed via RAID drivers that set aside a certain number of sectors for their own purposes. This works fine *IF* everything using the disk accesses it the same way. If, however, some tools access the "raw" version of the disk and others access it via the modified view, then you get inconsistencies like those being reported here. In most such cases, the solution is to disable the RAID features by disabling the RAID options in the firmware and/or by uninstalling RAID software or reconfiguring RAID settings under Linux.

The puzzling thing about this is that there seems to be an inconsistency in the way gdisk and fdisk are viewing the disk, since the two programs use the same system calls to determine the disk's size. (Or they did at one time; it's conceivable their code has diverged over time.) This could be explained if the two were fed different device identifiers to access the disk. The bug report is vague on this detail. It's also conceivable, if gdisk and fdisk have diverged in the way they identify the disk size, that there's a bug in one or the other utility or that there's a kernel or driver bug that's feeding inconsistent data via the two kernel calls.

FWIW, gdisk does *NOT* use data in the protective MBR to determine the total disk size, as the original bug report suggests; gdisk uses the disk size as reported by the kernel using Linux system calls, just as fdisk does.

Given that Mac OS X's Disk Utility and GParted under Ubuntu 10.04 both work fine, I think it's very likely that this is an issue with an errant RAID configuration or a kernel or driver bug in more recent versions of Ubuntu. (All versions of GParted are very fussy about partition table errors, typically showing a blank disk when they run into problems, so if GParted under Ubuntu 10.04 worked, then that means it did not run into a size inconsistency.)

[Ahmad Moawya (ahmadmoawya)]

Many thanks for you time and help.
Hope to see this fixed soon.

[Phillip Susi (psusi)]

gdisk and fdisk do not appear to be disagreeing on the size of the disk. I reproduced the problem in an image file by restoring the gpt backup and gdisk does not complain about the size mismatch when I make the image file 312579695 sectors long, but it does complain that the disk has the wrong size when I made the image file 312581808 sectors long. This points to gdisk comparing the disk size to the MBR size, rather than the GPT size.

Ahmad, could you attach your /var/log/kern.log file, and also post the output of sudo hdparm -I /dev/sda? I'd like to try and verify what the correct size of your drive is. Also you might try booting into MacOS and and see how big it thinks the drive is, and if you still have a 10.04 cd, boot it and check fdisk -l there, and make sure parted still does not crash.

I suspect that the MBR size got changed since you last ran 10.04, and trying it again will now crash just like 12.04 does, and MacOS does not report anything wrong because it doesn't bother checking the size, or because it is checking the wrong size, like gdisk is.

[Ahmad Moawya (ahmadmoawya)]

Here is the output of sudo hdparm -I /dev/sda:

root@ubuntu:~# hdparm -I /dev/sda

/dev/sda:

ATA device, with non-removable media
 Model Number: SAMSUNG HD161HJ
 Serial Number: S0V3JDWQ610766
 Firmware Revision: GF100-07
 Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5
Standards:
 Used: ATA-8-ACS revision 3b
 Supported: 8 7 6 5
Configuration:
 Logical max current
 cylinders 16383 16383
 heads 16 16
 sectors/track 63 63
 --
 CHS current addressable sectors: 16514064
 LBA user addressable sectors: 268435455
 LBA48 user addressable sectors: 312579695
 Logical/Physical Sector size: 512 bytes
 device size with M = 1024*1024: 152626 MBytes
 device size with M = 1000*1000: 160040 MBytes (160 GB)
 cache/buffer size = 8192 KBytes (type=DualPortCache)
Capabilities:
 LBA, IORDY(can be disabled)
 Queue depth: 32
 Standby timer values: spec'd by Standard, no device specific minimum
 R/W multiple sector transfer: Max = 16 Current = 16
 Advanced power management level: 254
 Recommended acoustic management value: 254, current value: 0
 DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6 udma7
      Cycle time: min=120ns recommended=120ns
 PIO: pio0 pio1 pio2 pio3 pio4
      Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
 Enabled Supported:
    * SMART feature set
      Security Mode feature set
    * Power Management feature set
    * Write cache
    * Look-ahead
    * Host Protected Area feature set
    * WRITE_BUFFER command
    * READ_BUFFER command
    * NOP cmd
    * DOWNLOAD_MICROCODE
    * Advanced Power Management feature set
      SET_MAX security extension
      Automatic Acoustic Management feature set
    * 48-bit Address feature set
    * Device Configuration Overlay feature set
    * Mandatory FLUSH_CACHE
    * FLUSH_CACHE_EXT
    * SMART error logging
    * SMART self-test
    * General Purpose Logging feature set
    * 64-bit World wide name
    * {READ,WRITE}_DMA_EXT_GPL commands
    * Segmented DOWNLOAD_MICROCODE
    * Gen1 signaling speed (1.5Gb/s)
    * Gen2 signaling speed (3.0Gb/s)
    * Native Command Queueing (NCQ)
    * Host-initiated interface power management
    * Phy event counters
      DMA Setup Auto-Activate optimization
      Device-initiated interface power management
    * Software settings preservation
    * SMART Command Transport (SCT) feature set
    * SCT Long Sector Access (AC1)
    * SCT LBA Segment Access (AC2)
    * SCT Error Recovery Control (AC3)
    * SCT Features Control (AC4)
    * SCT Data Tables (AC5)
Security:
 Master password revision code = 65534
  supported
 not enabled
 not locked
 not frozen
 not expired: security count
  supported: enhanced erase
 54min for SECURITY ERASE UNIT. 54min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50000f00db610766
 NAA : 5
 IEEE OUI : 0000f0
 Unique ID : 0db610766
Checksum: correct

I'll post the rest of the stuff in a little while :)

[Ahmad Moawya (ahmadmoawya)]

Here's the kernel log for the HDD detection:

Aug 7 18:43:03 ubuntu kernel: [ 4.308238] ata1.00: HPA detected: current 312579695, native 312581808
Aug 7 18:43:03 ubuntu kernel: [ 4.308244] ata1.00: ATA-8: SAMSUNG HD161HJ, GF100-07, max UDMA7
Aug 7 18:43:03 ubuntu kernel: [ 4.308247] ata1.00: 312579695 sectors, multi 16: LBA48 NCQ (depth 0/32)
Aug 7 18:43:03 ubuntu kernel: [ 4.316237] ata1.00: configured for UDMA/133
Aug 7 18:43:03 ubuntu kernel: [ 4.316379] scsi 0:0:0:0: Direct-Access ATA SAMSUNG HD161HJ GF10 PQ: 0 ANSI: 5
Aug 7 18:43:03 ubuntu kernel: [ 4.316540] sd 0:0:0:0: [sda] 312579695 512-byte logical blocks: (160 GB/149 GiB)
Aug 7 18:43:03 ubuntu kernel: [ 4.316585] sd 0:0:0:0: Attached scsi generic sg0 type 0
Aug 7 18:43:03 ubuntu kernel: [ 4.316607] sd 0:0:0:0: [sda] Write Protect is off
Aug 7 18:43:03 ubuntu kernel: [ 4.316610] sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
Aug 7 18:43:03 ubuntu kernel: [ 4.316636] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
Aug 7 18:43:03 ubuntu kernel: [ 4.328993] sda: unknown partition table
Aug 7 18:43:03 ubuntu kernel: [ 4.329204] sd 0:0:0:0: [sda] Attached SCSI disk

[Ahmad Moawya (ahmadmoawya)]

Here's fdisk output under OS X:

Disk: /dev/rdisk0 geometry: 19457/255/63 [312579695 sectors]
Signature: 0xAA55
         Starting Ending
 #: id cyl hd sec - cyl hd sec [ start - size]
------------------------------------------------------------------------
 1: EE 0 0 2 - 1023 255 63 [ 1 - 312579694] <Unknown ID>
 2: 00 0 0 0 - 0 0 0 [ 0 - 0] unused
 3: 00 0 0 0 - 0 0 0 [ 0 - 0] unused
 4: 00 0 0 0 - 0 0 0 [ 0 - 0] unused

Disk utility in MacOS reports this:

Total Size: 160.0 GB (160040803840 Bytes) (exactly 312579695 512-Byte-Blocks)

[Phillip Susi (psusi)]

Looks like that Host Protected Area feature again... your drive has been configured to lie about its size and claim it is slightly smaller than it really is. The MBR contains the smaller size, but the GPT contains the larger size. It is possible that the HPA was added after you created the GPT, and some partition tool updated the MBR to the new, smaller size, but not the GPT.

[Ahmad Moawya (ahmadmoawya)]

According to OS X, the number of sectors is 312579695 in both diskutil (GPT) and fdisk (protective MBR),
So, although the actual GPT contains the larger size, OS X diskutil respects the HPA and lists the smaller size.

So if I understood correctly, the problem here is that linux is not respecting the HPA and therefore confusing partitioning apps with 2 different sector numbers for the same disk?! Kernel issue?

[Ahmad Moawya (ahmadmoawya)]

After some searching, I found that the kernel team has indeed removed support for disabling HPA.

check this for example:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=578272

[Rod Smith (rodsmith)]

Psusi wrote: "This points to gdisk comparing the disk size to the MBR size, rather than the GPT size."

I assure you that this is **NOT** the case! The code that computes the disk size does **NOT** refer to MBR data structures. GPT fdisk works fine on over-2TiB disks, for which MBR data structures are necessarily much smaller than the disk's actual size. It also works fine with hybrid MBRs, which have 0xEE partitions in the MBR that are generally much smaller than the disk's size, and additional partitions that may or may not pad out the length. You're seeing a correlation and inferring causation -- always a risky endeavor!

Ahmad Moawya wrote: "I verified my disk using GPT fdisk (gdisk) and verification was successful (found valid GPT and protective MBR)"

When I first read this statement, I assumed you meant that you used the gdisk verify ("v" from any menu) function, but it occurs to me that you might have meant that you launched gdisk and saw the "partition table scan" results that said the GPT was valid. If so, please run gdisk again and do a "v" on the disk. Also, check the gdisk output when you type "p" to see what it reports as the disk's size in sectors. My suspicion is that gdisk will report a disk size of 312579695 sectors and, when you do a "v", it will report errors relating to the disk being too small for the partitions it contains. If so, then it would appear that the HPA feature is carving out part of the disk, as psusi suggests, and that all your current tools are using this HPA-reduced size, but that the disk was originally prepared with software (a Linux or other OS kernel) that didn't respect the HPA, or with HPA settings set at some other value. With the HPA now active, the fdisk output is largely irrelevant but partially diagnostic, OS X's Disk Utility isn't reporting an error because it's fairly lax about reporting errors in the placement of the backup data, parted is throwing up its hands as it usually does with damaged disks, and gdisk is reporting an error only when told to run a diagnostic (or it would report an error if you told it to save the partition table). That last is speculative, though; the gdisk data you've provided is limited.

I have limited experience with HPA, but according to Wikipedia (http://en.wikipedia.org/wiki/Host_protected_area), there are various tools you can use to manipulate it. You should research this more, but one possible fix would be to use one of those tools to disable HPA. Another fix would be to use the "e" option on gdisk's experts' menu to move the backup GPT data to where it belongs (given the HPA-reduced size) and then re-save the partition table. It should then be possible to use libparted-based tools on the disk.

Of course, even if I'm right about all this, that doesn't mean that libparted shouldn't handle such errors better.

[Ahmad Moawya (ahmadmoawya)]

@Roderick:
Sorry, I didn't give enough details. I did use the v option to verify and got the following:

No problems found. 526373 free sectors (257.0 MiB) available in 3
segments, the largest of which is 264223 (129.0 MiB) in size.

and with the p command, I got:

Disk /dev/disk0: 312579695 sectors, 149.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 777989DA-9962-2B05-18B6-4865BE87B817
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 312581774
Partitions will be aligned on 8-sector boundaries
Total free space is 526373 sectors (257.0 MiB)

Number Start (sector) End (sector) Size Code Name
   1 40 409639 200.0 MiB EF00
   2 409640 42101263 19.9 GiB AF00 Apple_HFS_Untitled_2ae4
   3 42363408 312317551 128.7 GiB AF00

I think that after all this reading and testing, it's safe to assume that gdisk is off the hook, and by that I mean its bug-free :D

BTW, I'd like to use this chance to thank you for the amazing gdisk tool you made. It's probably the most advanced GPT disk utility available.

@Everyone, I think we can all agree that whether the kernel is handling the HPA correctly or not, parted must be patched to better handle this scenario.

For the mean time, I'll be flagging the gdisk bug as invalid (till an admin deletes it) since I now believe it's actually handling the disk correctly and the problem is basically that parted cannot handle the disk without the kernel disabling HPA (like older kernel did in 10.04).

[Phillip Susi (psusi)]

Actually it is the reverse Ahmad. 10.04 disabled the HPA, making use of the full disk. Thus, parted compared the larger, true disk size with what the GPT claimed, and was happy since they agreed. 12.04 respects the HPA like MacOS and Windows do ( unless it detects a partition that claims to use the protected space, then it disables the HPA to allow access to that partition ). This causes the GPT size to disagree with the HPA limited size the kernel uses, and so parted barfs. In this situation, parted *should* issue a warning and offer to fix the error.

I too was able to run the gdisk verify command on an image with the smaller size without complaint so while it is possible my assumption about gdisk is wrong, it is clear that it is doing *something* wrong since it should be complaining when the image is the smaller size, rather than when it is the larger size. Looking at the sizes in both the GPT and the MBR, only the MBR seemed to agree with the smaller size, so it seemed a good assumption that gdisk was using that size.

[Ahmad Moawya (ahmadmoawya)]

Thanks for the explanation, Phillip.
Hope I don't sound pushy, but when can I expect a fix for this issue?

[Phillip Susi (psusi)]

Probably not any time soon, which is why I suggested repartitioning the disk to fix it.

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Karol Bryd (karolbe)]

This bug is causing Ubuntu Ubiquity installer to crash.

[Oliver Joos (oliver-joos)]

I see this crash with an external USB disk with MBR only (no GPT!)

I tried to shrink a FAT32 partition with gparted when the crash occurred the first time. Now the partition still has its old size, but the FAT32 filesystem has the new, lower size. Fortunately no data was lost.

IMO the importance of this bug is not low.

[Phillip Susi (psusi)]

If you aren't using gpt then you aren't getting the error message and thus aren't related to this bug. That message again was:

Assertion (last_usable <= disk->dev->length) at ../../../libparted/labels/gpt.c:718 in function _parse_header() failed.

[Phillip Susi (psusi)]

It seems the bug on the gdisk side was fixed sometime between 0.5.1 and 0.8.1: the latter version now correctly reports the size error.