clean_up_caches needs to better account for image converting

Bug #1356362 reported by Tom Cammann
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
Medium
Yuriy Zveryanskyy

Bug Description

When rebuilding and deploying my overcloud repeatedly on devtest the undercloud has been running out of disk space on /dev/sda1, the images in /mnt/state/var/lib/ironic/master_images are not being cleaned up when deploying a new overcloud.

I have been manually deleting them before running up my overcloud again. When /dev/sda1 runs out of space devtest ends up freezing and getting into a unstable state.

To reporduce:

Using devtest:

With the correct env settings and in tripleo-incubator/scripts

./devtest.sh # will take a while
source ../undercloudrc
# you can now check /mnt/state/var/lib/ironic/master_images for the cached images in the undercloud
heat stack-delete overcloud
./devtest_overcloud.sh
# let the overcloud try come up again and then check the undercloud master_images directory for cached images.

Running the overcloud up for this 2nd time fails in ironic.

/var/log/upstart/ironic-conductor.log:Command: qemu-img convert -O raw /mnt/state/var/lib/ironic/master_images/tmpVXkZmr/3f5b8944-9a1b-4a19-a67f-7b713e8e319c.part /mnt/state/var/lib/ironic/master_images/tmpVXkZmr/3f5b8944-9a1b-4a19-a67f-7b713e8e319c.converted
Exit code: 1
Stdout: ''
Stderr: 'qemu-img: error while writing sector 3017344: No space left on device\n'

Tags: pxe
Changed in ironic:
status: New → Confirmed
Revision history for this message
Dmitry Tantsur (divius) wrote :

Not clearing images is not a bug - we maintain a local cache. Bug is that we have a bad detection of low disk space. As a workaround you can reduce cache size via option pxe/image_cache_size.

summary: - Cleanup not clearing master_images
+ clean_up_caches needs to better account for image converting
Changed in ironic:
status: Confirmed → Triaged
importance: Undecided → Medium
tags: added: pxe
Changed in ironic:
assignee: nobody → Yuriy Zveryanskyy (yzveryanskyy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/115974

Changed in ironic:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.openstack.org/115974
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=d7cdc75841b729b274bf816fc64382aa963c1704
Submitter: Jenkins
Branch: master

commit d7cdc75841b729b274bf816fc64382aa963c1704
Author: Yuriy Zveryanskyy <email address hidden>
Date: Thu Aug 21 10:35:49 2014 +0300

    Control extra space for images conversion in image_cache

    Size of image converted to raw format can be growing up to virtual_size.
    This patch adds new _fetch_to_raw function to image_cache which try to
    free extra space for image conversion if needed.

    Closes-Bug: #1356362
    Change-Id: I69712b078113ef0640859389ef4ab9039135f612

Changed in ironic:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in ironic:
milestone: none → juno-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in ironic:
milestone: juno-3 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.