Enable creation of out-of-order partition tables to make Windows-interoperable USB disks

[Daniel Richard G.]

Binary package hint: partman-base

This is a wishlist item, filed at the suggestion of Colin Watson in a thread on the grub-devel mailing list:

    http://lists.gnu.org/archive/html/grub-devel/2010-01/msg00202.html

When partitioning a large external USB hard drive on which to install Ubuntu, there are two competing considerations that the user may have in mind:

1. Some computers will not be able to boot off the USB hard drive if /boot is farther than a certain distance from the start of the disk (i.e. a BIOS boot barrier, a problem which still plagues us nowadays). Therefore, /boot (or "/", if not separate) should be the first/first-ish partition, with the bulk of the disk after it.

2. When a USB disk is connected to a Windows system, Windows will mount the first partition, and ignore any others on the disk. So if the bulk of the disk is a large FAT32/NTFS partition, then that should be the first partition.

A nifty way of resolving the conflict between these two is to use an out-of-order partition table. Partition #1 is the large FAT32/NTFS partition, but in the disk's physical layout, it is actually the last partition. Partition #2 is a Linux partition, and is first in the disk's layout. Partitions #3, #4 et al. occupy the space between the end of partition #2 and the beginning of partition #1. An illustrated example:

    |<-sdx2->|<--sdx3-->|<--sdx4-->|<------------sdx1------------>|

Windows will correctly mount and use the "first" partition, and the Linux boot process (i.e. GRUB2) will sidestep any BIOS limitations because it doesn't need to read far into the disk to find /boot.

Currently, however, there is no way to produce a partition table like this in the Ubuntu graphical partitioner. The wish is to make that possible.

Caveat: Colin mentioned that GRUB2 should eventually have a module (ata.mod) that would allow its MBR-resident portion to find /boot using its own disk routines and not those of the BIOS, thereby avoiding any potential BIOS limitations. For the time being, however, ata.mod appears not to be stable enough for production use. (And for my part, I'm not sure whether it'll also be able to do the USB interfacing.)

[codeslinger (codeslinger)]

an interesting concept...

apparently in the one very specific scenario that you describe, this works ok.

However, beware, that I once had a disk with multiple partitions on it with a mixture of windows and linux.

due to splitting one partition into two, I ended up with an out of order partition table (logical)

the result is that when I booted into windows... it trashed the entire hard disk... it did not like having the out of order partition and so decided to replace the partition table with one that it liked.

on the other hand, I see oems shipping partition tables that have been deliberatly corrupted by overlapping them and this seems to work... but those partitions are not out of order, just overlapped to prevent windows from seeing it, typically a repair/tools partion that they want hidden from windows, but bootable.

anyway, bottom line is that out of order partitions entails substantial risk of disk corruption if the disk is accessed by windows.

[Daniel Richard G. (skunk)]

Codeslinger, Windows is not known to "trash" hard drives when booting for no other reason than seeing an out-of-order partition table. I don't know what happened in your scenario, but it was likely some errant tool like a defragmenter or partitioner that was not able to handle such a table. If Windows doesn't like a partition table, the standard behavior is that it doesn't recognize the drive, let alone boot.

[codeslinger (codeslinger)]

this occured with windows 2000, and I am pretty sure that it happened exactly as described, but that was awhile ago I might have misremembered something. It probably also depends on the specific sequence of the partitions.

Windows 2000 has a very finicky file system. These days winows 2000 is a lot less relevant but believe it or not it still gets used due to the costs and incompatibilites with legacy apps.

Perhaps other versions of MS Windows don't have this problem. However out-of-order partition tables are actually quite rare so I doubt very much that Microsoft spends significant time testing this scenario much less fixing any bugs found (MS Word shipped with 20000+ documented wontfix bugs because they weren't "common enough scenarios", as decided by them, even though some of those bugs caused crashes and corruption of documents).

If MS Windows is not known to have problems with out-of-order tables perhaps it's because people avoid creating them... ;-)

By the key point I am making is that any such partitioning scheme will need careful and extensive testing with multiple os versions and types. The complexity of the testing needed to ensure safe operation could exceed the effort to write the actual code.

I do like the concept, but it may be safer to put the effort into enabling GRUB to bypass the BIOS limitation.