Glance doesn't clean up the partial uploaded image when there is no disk space

[Jenny Shieh]

When the Glance repository has no enough disk space, uploading image task failed with "413 Request Entity Too Large". Besides returning an error, it should also clean up the partial uploaded image at the disk. Otherwise, we need to manually clean it up.

[Brian Waldon (bcwaldon)]

Yeah, that would be a nice thing to do.

[Tim Spriggs (tims-t)]

Related to this, when swift encounters an error writing to its backend it will return a 503 to glance and the glance client will write something like this to the user:

Error uploading image: (BackendException): Failed to add object to Swift. Got error from Swift: put_object('glance', '6d5381e6-8295-40dd-b814-ce9c032937ef', ...) failure and no ability to reset contents for reupload.

In my case, I have two swift nodes that appear to have MTU issues and so they each get a partial upload and then timeout once a frame of the magic size stalls the connection. I can reproduce this reliably and now I have a nice mess to cleanup in the swift backend.

For all that run into this and come across the bug report, this bash snippet may be of interest (assuming bash version 4 and above):

---
#!/bin/bash

SWIFT_GLANCE_CONTAINER=${SWIFT_GLANCE_CONTAINER:=glance}

declare -A in_glance
for id in $(glance index | awk '{print $1}'); do in_glance[$id]=1; done
for id in $(swift list $SWIFT_GLANCE_CONTAINER); do
    [[ -z ${in_glance[$id]} ]] && \
    echo swift delete $SWIFT_GLANCE_CONTAINER $id
done
---

NB: You'll need to run with appropriate OS_AUTH_USER/OS_AUTH_URL/... variables set to view the entire glance image list as well as ST_AUTH/ST_KEY/ST_USER variables to view (and perhaps cleanup) the swift backend.

This won't actually delete anything but it will print out a list of "swift delete" commands that will do the job. If the output looks reasonable then simply remove the 'echo' and of course, YMMV...

[Brian Waldon (bcwaldon)]

So fixing this generically depends on a lot more work. I'm going to fix it specifically for the filesystem driver but punt on fixing it for other drivers until Grizzly. I'll file a new bug for that work.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/11805

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/11805
Committed: http://github.com/openstack/glance/commit/37d1c1d83ca7b6cc811c9fbf9e7012d54bc271bb
Submitter: Jenkins
Branch: master

commit 37d1c1d83ca7b6cc811c9fbf9e7012d54bc271bb
Author: Brian Waldon <email address hidden>
Date: Wed Aug 22 09:12:15 2012 -0700

    Remove partial image data when filesystem is full

    When an image is uploaded that fills up the local filesystem, attempt
    to delete the created file. This only applies to the 'filesystem'
    store driver.

    Fixes bug 1019421

    Change-Id: Ie12d736f3efb29b0862fef2899a1d2c142def38e