Images that are > 3G in size fail on a system with enough RAM for tmpfs

Bug #1289582 reported by Clint Byrum
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Fix Released

Bug Description

The heuristic used to check whether tmpfs can be used does not take into account that the image may be bigger than DIB_MIN_TMPFS GB. Once the image goes over around DIB_MIN_TMPFS*0.5, the final step that converts it to a raw disk image via loopback will fail because the tmpfs will fill up while copying everything onto the loopback.

dib should check for enough space left on the tmpfs to also store the loopback file, and if there isn't enough, build the loopback in a new non-tmpfs temp directory.

Revision history for this message
Clint Byrum (clint-fewbar) wrote :

Critical because this is breaking TripleO CI.

Revision history for this message
Robert Collins (lifeless) wrote :

I thought we did a mv, precisely to avoid this failure mode :(

Revision history for this message
Robert Collins (lifeless) wrote :

I've made the nova flavor for h1.large 16G memory, see if this fixes it.

Revision history for this message
Robert Collins (lifeless) wrote :

We're getting green tests so that does appear to be a valid workaround

Changed in diskimage-builder:
importance: Critical → High
Revision history for this message
Clint Byrum (clint-fewbar) wrote :

FYI, the reason mv doesn't fix the problem entirely is that there is no synchronous connection between the mv and the growth of the loopback file. One hypothesis is that ext4 extents cause a large section of the file to be pre-allocated all at once.

It kind of doesn't matter though because the heuristic also just assumes that images will be ~ 2G.. which is clearly false.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to diskimage-builder (master)

Submitter: Jenkins
Branch: master

commit 3fada2566a366c8bc1b668e91c8412b8fb63fd7a
Author: Derek Higgins <email address hidden>
Date: Thu May 1 10:22:15 2014 +0100

    Build raw image in separate tmpfs

    If the tmpfs being used to hold the image filesystem is close to being
    full then housing the raw disk image in the same place can be
    problematic as it tends to grow faster then the filesystem shrinks when
    data is being moved into it.

    Putting them both into separate tmpfs's will allow there to be an
    overlap where they jointly use more then the size limit for a singe tmpfs.

    Change-Id: Ia17ca357d3b865d70a2d3e13e0479d008ca5f924
    Closes-Bug: #1289582

Changed in diskimage-builder:
status: Triaged → Fix Committed
Changed in diskimage-builder:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.