Support "fudge factor" for file system sizing

Bug #1632085 reported by Barry Warsaw on 2016-10-10
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ubuntu Image
Wishlist
Barry Warsaw

Bug Description

We already add several fudge factors when calculating the rootfs and bootfs sizes. We always increase the size by a factor of 1.5 after looking at the sizes of all the files in the source directory, then for the rootfs, we add an additional 8MiB to account for ext4 overhead, with this comment in calculate_rootfs_size():

    def calculate_rootfs_size(self):
        # Calculate the size of the root file system. Basically, I'm trying
        # to reproduce du(1) close enough without having to call out to it and
        # parse its output.
        # On a 100MiB filesystem, ext4 takes a little over 7MiB for the
        # metadata. Use 8MiB as a minimum padding here.
        self.rootfs_size = self._calculate_dirsize(self.rootfs) + MiB(8)

Ogra wants to be able to define some additional "wiggle room" using a cli option or envar. His use case is that kvm images need some extra space.

I think a better way might be to specify the additional fudge factor in the gadget.yaml. A strawman proposal is to add a 'rootfs-size' option (and possibly a bootf-size option) to the spec, which could take various forms, e.g. to handle Ogra's case:

rootfs-size: x1.5+500

This would live in the volume section. Format TBD. Is this a good idea?

Barry Warsaw (barry) wrote :

Oli makes the case for a command line switch, from the mailing list:

"""
defining it in the gadget would be rather awful since that means if we
use an identical image for cloud, VM  and "normal PC install" we would
need a gadget per image just for that one difference ... (the pc image
that you can use for all three consists of: pc (gadget, pc-kernel
(obviously kernel) and core (rootfs).

also not every cloud provider or VM user might want to use the same
basic size so they would also need their own gadgets that set the image
size here, i think that makes it to complex, thus i would rather go
with a commandline option to the build tool.
"""

Barry Warsaw (barry) wrote :

Motivating use case, also from Oli on the mailing list:

"""
on VMs the img file defines the "physical VM disk" while for an actual
install you write the img file to a pyhsical disk device ... the resize
code will find that the partition table does not cover all of the disk
before first boot and start the resize process so here the img size
wont be taken into account at all.

on a VM, all the resize code will see is the img file that pretends to
be a harddisk and it will find there is nothing to resize to.
"""

I plan on supporting this through a command line option that overrides the hardcoded 8MiB rootfs fudge factor.

For now, I won't change the 1.5x size calculation boost on all directory calculations, although this should certainly be addressed later.

no longer affects: snappy
Changed in ubuntu-image:
assignee: nobody → Barry Warsaw (barry)
status: New → In Progress
Barry Warsaw (barry) wrote :

Note that this is different than LP: #1632424

Steve Langasek (vorlon) wrote :

From the mailing list:

I don't think a 'fudge factor' option is particularly clean or intuitive. I
think what you actually want is to be able to declare the overall size of
the image that you're building. It's a waste of mental energy to think
about VM image sizes in increments less than 1G, especially given that these
are all still going to be sparse files.

So I would suggest:

 - a --size option to ubuntu-image to specify the full image size (needs
   some careful thought for the multiple-image case)
 - keep the 50% buffer on ext4 filesystem size as a floor (the '50%' can be
   refined over time with investigation into the actual size requirements
   for ext4 metadata, but this is not a high priority; highest priority is
   that it works reliably)
 - if --size is smaller than the minimum size needed to accommodate the
   contents, warn but build the image anyway - because all the expensive
   work of ubuntu-image happens /before/ we're able to calculate the final
   size, so throwing that all away at the end would be /very annoying/.

Barry Warsaw (barry) wrote :

I want to be clear about one point: the size that this switch will control is the overall disk image size, not the size of any individual partition within the image. I.e. it's the size of the resulting file written to via the --output switch.

Thus all the discussion about fudge factors is off-topic, as those are applied to partition sizes. We will address those separately.

Here's how the image size is calculate (in prepare_filesystems()):

        self.image_size = ceil((self.rootfs_size + next_avail) /
                               1024 + 17) * 1024

I propose therefore to call the switch `--image-size` (no shortcut) which takes a value of the same format as the other size parameters in the gadget.yaml. A bare integer means bytes, and it accepts suffixes 'M' for MiB and 'G' for GiB.

Since u-i only handles single volume gadget.yamls atm, --image-size=10G would refer to the first and only image. In the future we can relax the "only" part, so that an unqualified value would refer to first volume. This would be equivalent to the (future) extended syntax --image-size=0:10G. Later we can accept multiple --image-size arguments, and/or a syntax such as --image-size=0:10G,1:1000M,2:1073741824

We'll check that --image-size is at least as big as the calculated size. I'd prefer to fail hard in that case, since it's generally easier to relax a constraint than to tighten it, but as Steve points out, that would throw away a lot of work and even though we support preserving the working directory, we don't currently have a way to resume after failure. So, if the --image-size is smaller than the calculated size, we'll warn and ignore it, going with the calculated size instead. That way, we'll preserve all of the work and won't generate a corrupted image.

Barry Warsaw (barry) wrote :

Alternative to the indexes, we could support named volumes.

Barry Warsaw (barry) on 2016-10-14
Changed in ubuntu-image:
status: In Progress → Fix Committed
milestone: none → 0.8
Barry Warsaw (barry) on 2016-10-17
Changed in ubuntu-image:
status: Fix Committed → Fix Released
Martin Pitt (pitti) wrote :

Please add an SRU test case and regression potential.

Changed in ubuntu-image (Ubuntu Zesty):
status: New → Fix Committed
Barry Warsaw (barry) on 2016-10-18
no longer affects: ubuntu-image (Ubuntu)
no longer affects: ubuntu-image (Ubuntu Yakkety)
no longer affects: ubuntu-image (Ubuntu Zesty)
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers