resize2fs -P minimum size differs greatly between v1.42.10 and v1.42.9

[Scott Moser]

I have a disk image created as part of lp:maas-images.
This process takes a cloud image (http://cloud-images.ubuntu.com) such as
http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64.tar.gz .
Inside that .tar.gz is a .img file. The .img file is operated on
by loop mount, chroot, apt-get install and other changes.

Immediately upon download and extraction of a pristine image, resize2fs output
on trusty, utopic and vivid differ. A summary of differences:
 a.) 'resize2fs -P' on
     trusty (1.42.9) : 230329
     utopic (1.42.10): 236103
     vivid (1.42.12): 236103
 b.) vivid resize2fs requires running 'e2fsck -f' first.

I do realize that 'resize2fs -P' reports "Estimated minumum size" and that a
difference of ~6k blocks doesn't seem like too big of a deal, but I'm seeing
this behavior on a much grander scale after doing some loop mount operations
on the disk image as part of the maas-images build process (lp:maas-images).
On that derived image, I see 'resize2fs -P' numbers like:
 trusty: 274500
 utopic: 358400
 vivid: 358400
The 83900 blocks out of a 358400 seems significant (23% of the fs size),
and results in the image appearing full on utopic/vivid, even though
a mount and df show only 74% full.

See below for more information.

## The original image as downloaded.
$ wget http://cloud-images.ubuntu.com/releases/14.04/release-20150123/ubuntu-14.04-server-cloudimg-amd64.tar.gz
$ tar -Sxvzf ubuntu-14.04-server-cloudimg-amd64.tar.gz
$ md5sum trusty-server-cloudimg-amd64.img
9fd4790c920aaeb0b503ebd2bf7f383c trusty-server-cloudimg-amd64.img

## Trusty
$ resize2fs -P trusty-server-cloudimg-amd64.img
resize2fs 1.42.9 (4-Feb-2014)
Estimated minimum size of the filesystem: 230329

$ dumpe2fs trusty-server-cloudimg-amd64.img | pastebinit
dumpe2fs 1.42.9 (4-Feb-2014)
http://paste.ubuntu.com/9898743/

## Utopic
$ resize2fs -P trusty-server-cloudimg-amd64.img
resize2fs 1.42.10 (18-May-2014)
Estimated minimum size of the filesystem: 236103

$ dumpe2fs trusty-server-cloudimg-amd64.img | pastebinit
dumpe2fs 1.42.10 (18-May-2014)
http://paste.ubuntu.com/9898789/

## Vivid
$ resize2fs -P trusty-server-cloudimg-amd64.img
resize2fs 1.42.12 (29-Aug-2014)
Please run 'e2fsck -f trusty-server-cloudimg-amd64.img' first.

$ cp trusty-server-cloudimg-amd64.img trusty-server-cloudimg-amd64.img.dist
$ e2fsck -f trusty-server-cloudimg-amd64.img
e2fsck 1.42.12 (29-Aug-2014)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
cloudimg-rootfs: 57885/90112 files (0.0% non-contiguous), 207837/360448 blocks

$ resize2fs -P trusty-server-cloudimg-amd64.img
resize2fs 1.42.12 (29-Aug-2014)
Estimated minimum size of the filesystem: 236103

$ md5sum trusty-server-cloudimg-amd64.img.dist trusty-server-cloudimg-amd64.img
9fd4790c920aaeb0b503ebd2bf7f383c trusty-server-cloudimg-amd64.img.dist
c38a3ca8cba7996c85365e122175d013 trusty-server-cloudimg-amd64.img

$ dumpe2fs trusty-server-cloudimg-amd64.img | pastebinit
dumpe2fs 1.42.12 (29-Aug-2014)
http://paste.ubuntu.com/9898831/

$ dumpe2fs trusty-server-cloudimg-amd64.img.dist | pastebinit
dumpe2fs 1.42.12 (29-Aug-2014)
http://paste.ubuntu.com/9898832/

## maas images image
## output below is for the maas build process output image
## This one specifically is an arm64 image, and the operations
## done (apt-get update) and such were done with the help chroot via
## qemu-static.

## build the image (works on trusty, fails due to resizing issues on utopic)
$ time maas-cloudimg2eph2 -vv --kernel=linux-generic --arch=arm64 \
  http://cloud-images.ubuntu.com/daily/server/vivid/20150125/vivid-server-cloudimg-arm64.tar.gz \
  /tmp/maas_final/vivid/arm64/20150125/root-image.gz \
  --krd-pack=linux-generic,/tmp/maas_final/vivid/arm64/20150125/vivid/generic/boot-kernel,/tmp/maas_final/vivid/arm64/20150125/vivid/generic/boot-initrd 2>&1 | tee out.log

$ pastebinit out.log
http://paste.ubuntu.com/9899165/

$ zcat /tmp/maas_final/vivid/arm64/20150125/root-image.gz > /tmp/root-image

$ ls -l /tmp/root-image
-rw-rw-r-- 1 ubuntu ubuntu 1468006400 Jan 27 14:56 /tmp/root-image

## see attached root-image.e2image.xz
$ e2image /tmp/root-image /tmp/root-image.e2image
e2image 1.42.9 (4-Feb-2014)
$ xz /tmp/root-image.e2image

## Now, the above 'root-image' file on utopic, vivid and trusty.
##
## Trusty
$ resize2fs -P /tmp/root-image
resize2fs 1.42.9 (4-Feb-2014)
Estimated minimum size of the filesystem: 274500

$ mkdir -p /tmp/mp && sudo mount -o loop,ro /tmp/root-image /tmp/mp &&
  df /tmp/mp && sudo umount /tmp/mp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop0 1356344 980284 359676 74% /tmp/mp

$ dumpe2fs /tmp/root-image | pastebinit
dumpe2fs 1.42.9 (4-Feb-2014)
http://paste.ubuntu.com/9899267/

## Utopic
$ resize2fs -P /tmp/root-image
resize2fs 1.42.10 (18-May-2014)
Estimated minimum size of the filesystem: 358400

$ mkdir -p /tmp/mp && sudo mount -o loop,ro /tmp/root-image /tmp/mp &&
  df /tmp/mp && sudo umount /tmp/mp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop0 1356344 980284 359676 74% /tmp/mp

$ dumpe2fs /tmp/root-image | pastebinit
dumpe2fs 1.42.10 (18-May-2014)
http://paste.ubuntu.com/9899313/

## Vivid
$ resize2fs -P /tmp/root-image
resize2fs 1.42.12 (29-Aug-2014)
Estimated minimum size of the filesystem: 358400

$ mkdir -p /tmp/mp && sudo mount -o loop,ro /tmp/root-image /tmp/mp &&
  df /tmp/mp && sudo umount /tmp/mp
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/loop0 1356344 980284 359676 74% /tmp/mp

$ dumpe2fs /tmp/root-image | pastebinit
dumpe2fs 1.42.12 (29-Aug-2014)
http://paste.ubuntu.com/9899328/

ProblemType: Bug
DistroRelease: Ubuntu 15.04
Package: e2fsprogs 1.42.12-1ubuntu1
ProcVersionSignature: User Name 3.18.0-9.10-generic 3.18.2
Uname: Linux 3.18.0-9-generic x86_64
ApportVersion: 2.15.1-0ubuntu2
Architecture: amd64
Date: Tue Jan 27 14:13:26 2015
Ec2AMI: ami-000000f5
Ec2AMIManifest: FIXME
Ec2AvailabilityZone: nova
Ec2InstanceType: m1.small
Ec2Kernel: aki-00000002
Ec2Ramdisk: ari-00000002
ProcEnviron:
 TERM=xterm
 PATH=(custom, no user)
 XDG_RUNTIME_DIR=<set>
 LANG=en_US.UTF-8
 SHELL=/bin/bash
SourcePackage: e2fsprogs
UpgradeStatus: No upgrade log present (probably fresh install)

[Scott Moser (smoser)]

To rule out kernel involvement, I took the same 'root-image' and ran this on utopic:

# these downloaded trusty versions [https://launchpad.net/ubuntu/+source/e2fsprogs]
$ sudo dpkg -i e2fslibs_1.42.9-3ubuntu1_amd64.deb
$ sudo dpkg -i e2fsprogs_1.42.9-3ubuntu1_amd64.deb

$ resize2fs -P /tmp/root-image
resize2fs 1.42.9 (4-Feb-2014)
Estimated minimum size of the filesystem: 274500

And then quickly checking that 'apt-get install e2fsprogs' installs utopic version 1.42.10 and:
$ resize2fs -P /tmp/root-image
resize2fs 1.42.10 (18-May-2014)
Estimated minimum size of the filesystem: 358400

[Chris J Arges (arges)]

I did a reverse bisect and found this to be the first 'bad' commit:
commit 45a78b88dba1db5758f68d0d3229d3cb42da6050
Author: Theodore Ts'o <email address hidden>
Date: Sun Apr 27 15:52:48 2014 -0400

    resize2fs: refine minimum required blocks for flex_bg file systems

[Theodore Ts'o (tytso)]

The change was deliberate, because under some extreme circumstances you could end up corrupting the file system image when resize2fs aborted in the middle of a shrink operation. This tends to make some users cranky when they lose their data.

If you had bothered to read the entire git commit description, you would have gotten a lot more context:

    resize2fs: refine minimum required blocks for flex_bg file systems

    The previous commit exposed bugs in the calculation for flex_bg file
    systems. The problem is that since (by default) we keep the metadata
    blocks for the flex_bg in the first block group of the flex_bg, and
    because we don't want to overwrite metadata blocks used by the
    original file system with data blocks make life easier in case the
    resize is aborted for some reason, we need to treat all of the
    metadata blocks in the existing flex_bg has in use for the purposes of
    calculate_minimum_resize_size().

    Even though this means we need to reserve more data blocks to avoid
    running out of space, the net result of these two commits is a net
    savings in how much we can shrink a file system.

    Using the following test sequence:

        mke2fs -F -t ext4 /tmp/foo.img 2T
        resize2fs -M /tmp/foo.img
        resize2fs -M /tmp/foo.img
        resize2fs -M /tmp/foo.img

    Here is the comparison in the resulting file systems between the old
    and new resize2fs (units are in 4k blocks):

                    resize #1 resize #2 resize #3
    old resize2fs 1117186 45679 43536
    new resize2fs 48784 37413 37392

    Signed-off-by: "Theodore Ts'o" <email address hidden>

Note that using resize2fs -M multiple times will result in a file system which is *not* optimized for speed. In fact, it can be really bad for slow devices, such as CD-ROM's and Amazon Elastic Block devices, since files and directories will end up getting fragmented and where the data blocks can end up getting placed farther than optimal from the inode table locations.

[Theodore Ts'o (tytso)]

Changing to invalid, will not fix upsteam.

[Scott Moser (smoser)]

Ted, thanks for your response, and sorry for the spam.

Can you just verify that this specific bit seems correct:
 | On that derived image, I see 'resize2fs -P' numbers like:
 | trusty: 274500
 | utopic: 358400
 | vivid: 358400
 | The 83900 blocks out of a 358400 seems significant (23% of the fs size),
 | and results in the image appearing full on utopic/vivid, even though
 | a mount and df show only 74% full.

Note, that :
  'df' reports the fs is 74% full,
  dumpe2fs reports the filesystem is 358400 blocks,
  resize2fs -P reports that the minimum size is 358400 blocks.

Ie, per resize2fs -P says we can't shrink this 1400M filesystem at all, which has greater than 300M available.

I'm willing to accept that this is academic, and an invalid bug, but I just wanted to clarify as the above seems wrong.