nova error handling causes glance to keep unlinked files open, wasting space
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Glance Client |
Fix Released
|
Medium
|
Unassigned | ||
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Chris Friesen |
Bug Description
When creating larger glance images (like a 10GB CentOS7 image), if we run into situation where we run out of room on the destination device, we cannot recover the space from glance. glance-api will have open unlinked files, so a TONNE of space is unavailable until we restart glance-api.
Nova will try to reschedule the instance 3 times, so should see this nova-conductor.log :
u'RescheduledEx
The problem is this code in nova.image.
if data is None:
return image_chunks
else:
try:
for chunk in image_chunks:
if close_file:
image_chunks is an iterator. If we take an exception (like we can't write the file because the filesystem is full) then we will stop iterating over the chunks. If we don't iterate over all the chunks then glance will keep the file open.
Changed in nova: | |
assignee: | nobody → Chris Friesen (cbf123) |
Changed in nova: | |
importance: | Undecided → Medium |
Changed in python-glanceclient: | |
status: | New → Fix Committed |
importance: | Undecided → Medium |
Changed in python-glanceclient: | |
milestone: | none → 1.0.0 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | none → liberty-3 |
status: | Fix Committed → Fix Released |
Changed in nova: | |
milestone: | liberty-3 → 12.0.0 |
Fix proposed to branch: master /review. openstack. org/188179
Review: https:/