nova error handling causes glance to keep unlinked files open, wasting space

[Chris Friesen]

When creating larger glance images (like a 10GB CentOS7 image), if we run into situation where we run out of room on the destination device, we cannot recover the space from glance. glance-api will have open unlinked files, so a TONNE of space is unavailable until we restart glance-api.

Nova will try to reschedule the instance 3 times, so should see this nova-conductor.log :
u'RescheduledException: Build of instance 98ca2c0d-44b2-48a6-b1af-55f4b2db73c1 was re-scheduled: [Errno 28] No space left on device\n']

The problem is this code in nova.image.glance.GlanceImageService.download():

        if data is None:
            return image_chunks
        else:
            try:
                for chunk in image_chunks:
                    data.write(chunk)
            finally:
                if close_file:
                    data.close()

image_chunks is an iterator. If we take an exception (like we can't write the file because the filesystem is full) then we will stop iterating over the chunks. If we don't iterate over all the chunks then glance will keep the file open.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/188179

[OpenStack Infra (hudson-openstack)]

Change abandoned by Chris Friesen (<email address hidden>) on branch: master
Review: https://review.openstack.org/188179
Reason: I think https://review.openstack.org/#/c/190359/ will be a better fix.

[Chris Friesen (cbf123)]

Not sure why it wasn't linked, but https://review.openstack.org/#/c/190359/ is being proposed as a fix to this issue.

[Davanum Srinivas (DIMS) (dims-v)]

 https://review.openstack.org/#/c/190359/ is merged.