growpart on disk larger than 2TB fails

[Scott Moser]

=== Begin SRU Information ===
[Impact]
If growpart is run on a disk that is larger than 2 Terabytes and an MBR
partition table, then the partition table will not be updated correctly.

The result is that the partition can actually be shrunk, which breaks the
filesystem on the disk.

[Test Case]
To test, you need a block device > 2TB or a file > 2TB.
Because of sparse files, you can easily create one on local filesystem.
Below, an actual block device can be used just as well.

## Create a file that is 2.5TB.
$ dev="/tmp/my.img"
$ truncate --size 2500G $dev

## if 'dev' is a file, then 'update' wont work. In upstream trunk it will
## not attempt to run partx on a file. update only exists in saucy or later.
[ -b "$dev" -a "$(lsb_release -sc)" != "precise" ] &&
   update_f="--update=auto" || update_f="--update=off"

## upstream cloud-utils, but broken
## use sfdisk to partition it with a 500GB first partition
## then show that that generally worked
$ echo "2048,$((500*2*1024*1024)),L," | sfdisk --unit S $dev
$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 1048578047 1048576000 83 Linux

## use growpart to grow the partition
## then verify it worked. We should see a ~2TB partition size
$ growpart ${update_f} /tmp/my.img 1
CHANGED: partition=1 start=2048 old: size=1048576000 end=1048578048 new: size=4294965248,end=4294967296
$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 4294967295 4294965248 83 Linux

## When this failed, it would fail like this:
## the new 'size' is > 2TB and thus invalid. And sfdisk -l would show
## the bad table.
$ growpart ${update_f} /tmp/my.img 1
CHANGED: partition=1 start=2048 old: size=1048576000 end=1048578048 new: size=5242874962,end=5242877010
$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 947909713 947907666 83 Linux

[Regression Potential]
Regression potential is little to none.
This was 100% broken with disks over 2TB previously.
The code now checks if the new size would be greater than 2TB. If it is, it
just limits the size to 2TB.

=== End SRU Information ===

For disks larger than 2TB (the amount addressable by MBR), growpart will mess up partition tables.
It also fails in such a way that it ends up shrinking the partition table, see below for an example.

In reality, this is probably an sfdisk bug, but I can work around it in growpart by just knowing the max size.

$ sudo growpart -v --dry-run /dev/sdb 1
geometry is -C 364801 -H 255 -S 63. total size=5860528065
max_end=5860528065 tot=5860528065 pt_end=209715200 pt_start=2048 pt_size=209713152
CHANGE: partition=1 start=2048 old: size=209713152 end=209715200 new: size=5860526017,end=5860528065
# === old sfdisk -d ===
# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start= 2048, size=209713152, Id=83, bootable
/dev/sdb2 : start= 0, size= 0, Id= 0
/dev/sdb3 : start= 0, size= 0, Id= 0
/dev/sdb4 : start= 0, size= 0, Id= 0
# === new sfdisk -d ===
# partition table of /dev/sdb
unit: sectors

/dev/sdb1 : start= 2048, size=5860526017, Id=83, bootable
/dev/sdb2 : start= 0, size= 0, Id= 0
/dev/sdb3 : start= 0, size= 0, Id= 0
/dev/sdb4 : start= 0, size= 0, Id= 0

$ sudo growpart /dev/sdb 1
CHANGED: partition=1 start=2048 old: size=209713152 end=209715200 new: size=5860526017,end=5860528065

$ grep "sdb" /proc/partitions
   8 16 2930266584 sdb
   8 17 782779360 sdb1

$ sudo sfdisk -l /dev/sdb -u -S
unrecognized format - using sectors

Disk /dev/sdb: 364801 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot Start End #sectors Id System
/dev/sdb1 * 2048 1565560768 1565558721 83 Linux
/dev/sdb2 0 - 0 0 Empty
/dev/sdb3 0 - 0 0 Empty
/dev/sdb4 0 - 0 0 Empty

ProblemType: Bug
DistroRelease: Ubuntu 12.04
Package: cloud-utils 0.25-0ubuntu5
ProcVersionSignature: User Name 3.2.0-57.87-generic 3.2.52
Uname: Linux 3.2.0-57-generic x86_64
ApportVersion: 2.0.1-0ubuntu17.6
Architecture: amd64
Date: Tue Dec 10 21:28:44 2013
MarkForUpload: True
PackageArchitecture: all
ProcEnviron:
 TERM=screen
 PATH=(custom, no user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: cloud-utils
UpgradeStatus: No upgrade log present (probably fresh install)

[Scott Moser (smoser)]

fixed in trunk at revision 250.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.27-0ubuntu5

---------------
cloud-utils (0.27-0ubuntu5) trusty; urgency=low

  * sync to trunk at revno 250
    * cloud-localds: make quiet by default (increase verbosity with '-v')
    * ubuntu-cloudimg-query: do not fail on no ami id found if no ami id is
    * necessary for the output requested (ie, allow 'armhf' queries of url)
    * growpart: fix bug when growing partitions on disks > 2TB. (LP: #1259703)
 -- Scott Moser <email address hidden> Wed, 11 Dec 2013 11:39:40 -0500

[Scott Moser (smoser)]

I went ahead and sent a mail about this upstream to util-linux mailing list at http://marc.info/?l=util-linux-ng&m=138678573111158&w=2 .
I stupidly forgot to put a subject.

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-utils into saucy-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-utils/0.27-0ubuntu4.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Brian Murray (brian-murray)]

Hello Scott, or anyone else affected,

Accepted cloud-utils into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/cloud-utils/0.25-0ubuntu5.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

[Scott Moser (smoser)]

=== Saucy verification ===
$ lsb_release -sc
saucy

$ echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" |
    sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update -q -y

$ sudo apt-get install -q -y cloud-guest-utils

$ apt-cache policy cloud-guest-utils
cloud-guest-utils:
  Installed: 0.27-0ubuntu4.1
  Candidate: 0.27-0ubuntu4.1
  Version table:
 *** 0.27-0ubuntu4.1 0
        500 http://archive.ubuntu.com/ubuntu/ saucy-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.27-0ubuntu4 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ saucy/main amd64 Packages

$ dev="/tmp/my.img"
$ truncate --size 2500G $dev
$ [ -b "$dev" -a "$(lsb_release -sc)" != "precise" ] &&
  update_f="--update=auto" || update_f="--update=off"
$ echo update_f=$update_f
update_f=--update=off

$ echo "2048,$((500*2*1024*1024)),L," | sfdisk --unit S $dev
#> removed output

$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 1048578047 1048576000 83 Linux

$ growpart ${update_f} $dev 1
CHANGED: partition=1 start=2048 old: size=1048576000 end=1048578048 new: size=4294965248,end=4294967296

$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 4294967295 4294965248 83 Linux

## now verify that still functions on real disk ~ 20G
$ dev=/dev/vdb
$ [ -b "$dev" -a "$(lsb_release -sc)" != "precise" ] &&
  update_f="--update=auto" || update_f="--update=off"

$ echo update_f=$update_f
update_f=--update=auto

$ echo "2048,$((10*2*1024*1024)),L," | sudo sfdisk --unit S $dev
#> removed output

$ sudo sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/dev/vdb1 2048 20973567 20971520 83 Linux

$ sudo growpart ${update_f} $dev 1
CHANGED: partition=1 start=2048 old: size=20971520 end=20973568 new: size=41940832,end=41942880

$ sudo sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/dev/vdb1 2048 41942879 41940832 83 Linux

[Scott Moser (smoser)]

=== precise verification ===

$ lsb_release -sc
precise

$ echo "deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main" |
    sudo tee /etc/apt/sources.list.d/proposed.list

$ sudo apt-get update -q -y && sudo apt-get install -q -y cloud-utils
$ apt-cache policy cloud-utils
cloud-utils:
  Installed: 0.25-0ubuntu5.1
  Candidate: 0.25-0ubuntu5.1
  Version table:
 *** 0.25-0ubuntu5.1 0
        500 http://archive.ubuntu.com/ubuntu/ precise-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     0.25-0ubuntu5 0
        500 http://nova.clouds.archive.ubuntu.com/ubuntu/ precise/main amd64 Packages

$ dev="/tmp/my.img"
$ truncate --size 2500G $dev
$ [ "$(lsb_release -sc)" = "precise" ] && update_f="" ||
  { [ -b "$dev" ] && update_f="--update=auto" || update_f="--update=off"; }
$ echo update_f=$update_f
update_f=

$ echo "2048,$((500*2*1024*1024)),L," | sfdisk --unit S $dev
#> removed output

$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 1048578047 1048576000 83 Linux

$ growpart ${update_f} $dev 1
CHANGED: partition=1 start=2048 old: size=1048576000 end=1048578048 new: size=4294965248,end=4294967296

$ sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/tmp/my.img1 2048 4294967295 4294965248 83 Linux

## now verify that still functions on real disk ~ 20G
$ dev=/dev/vdb
$ [ "$(lsb_release -sc)" = "precise" ] && update_f="" ||
  { [ -b "$dev" ] && update_f="--update=auto" || update_f="--update=off"; }
$ echo update_f=$update_f
update_f=

$ echo "2048,$((10*2*1024*1024)),L," | sudo sfdisk --unit S $dev
#> removed output

$ sudo sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/dev/vdb1 2048 20973567 20971520 83 Linux

$ sudo growpart ${update_f} $dev 1
CHANGED: partition=1 start=2048 old: size=20971520 end=20973568 new: size=41940832,end=41942880

$ sudo sfdisk --unit S -l $dev 2>/dev/null | grep "${dev}1"
/dev/vdb1 2048 41942879 41940832 83 Linux

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.25-0ubuntu5.1

---------------
cloud-utils (0.25-0ubuntu5.1) precise-proposed; urgency=low

  * fix growpart when disks operated on are larger than 2TB (LP: #1259703)
 -- Scott Moser <email address hidden> Wed, 11 Dec 2013 15:53:20 -0500

[Colin Watson (cjwatson)]

The verification of the Stable Release Update for cloud-utils has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

[Launchpad Janitor (janitor)]

This bug was fixed in the package cloud-utils - 0.27-0ubuntu4.1

---------------
cloud-utils (0.27-0ubuntu4.1) saucy-proposed; urgency=low

  * fix growpart when disks operated on are larger than 2TB (LP: #1259703)
 -- Scott Moser <email address hidden> Wed, 11 Dec 2013 15:18:42 -0500

[Scott Moser (smoser)]

fixed in 0.28