poor handling of download failure from glance in nova-compute

Bug #801412 reported by Jesse Andrews on 2011-06-24
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Pádraig Brady

Bug Description

Hypervisor: qemu/kvm
nova: trunk as of june 23
host os: maverick
type: multi-node install - nova + glance + openstack api

When nova-compute fails to download an image (for instance if you forget to set your glance_api_servers flag) it will create a 0 sized file:

root@356586-nova-compute-n01:/var/lib/nova/instances/_base# ls -al
drwxr-xr-x 2 nova nogroup 4096 2011-06-23 21:54 .
drwxr-xr-x 8 nova root 4096 2011-06-23 21:54 ..
-rw-r--r-- 1 nova nogroup 0 2011-06-23 21:47 00000001


Then on further attempts to launch it doesn't re-attempt to download.

Perhaps instead of downloading the file to the final location we should do a 00000001.download then move it upon finish?

Also checking the checksum before we finish the move?

Changed in nova:
importance: Undecided → High
Thierry Carrez (ttx) on 2011-06-27
Changed in nova:
importance: High → Medium
status: New → Confirmed
Jesse Andrews (anotherjesse) wrote :

If this is still an issue we should probably add this to essex. When this triggered it would cause all further launch this image to fail.

Changed in nova:
assignee: nobody → Pádraig Brady (p-draigbrady)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/3664
Committed: http://github.com/openstack/nova/commit/3db5b1e7099e1c10d557577350dc28e323c70af4
Submitter: Jenkins
Branch: master

commit 3db5b1e7099e1c10d557577350dc28e323c70af4
Author: Pádraig Brady <email address hidden>
Date: Thu Feb 2 14:56:54 2012 +0000

    fix stale libvirt images on download failure. Bug 801412

    If we've just truncated/created the file to receive
    the download, but the download fails, then delete the file.
    Otherwise subsequent downloads would not be attempted.

    Change-Id: Ibe9ba199b2f424d570ce7aedfb768ce283816a3b

Changed in nova:
status: In Progress → Fix Committed
Brian Waldon (bcwaldon) on 2012-02-02
Changed in nova:
milestone: none → essex-4

Fix proposed to branch: master
Review: https://review.openstack.org/4141

Changed in nova:
status: Fix Committed → In Progress

Reviewed: https://review.openstack.org/4141
Committed: http://github.com/openstack/nova/commit/46f7adadaf1d75f36a4544cb9c011056fd6d0e3a
Submitter: Jenkins
Branch: master

commit 46f7adadaf1d75f36a4544cb9c011056fd6d0e3a
Author: Pádraig Brady <email address hidden>
Date: Tue Feb 14 16:47:00 2012 +0000

    improve stale libvirt images handling fix. Bug 801412

    If the unlink fails (because the path wasn't
    even created for example), then that will shroud
    the original failure. Instead log failure to
    remove the stale image, and raise the original exception.

    Change-Id: I36c6968823bcf81d704319739b3a992dae75266a

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx) on 2012-02-29
Changed in nova:
status: Fix Committed → Fix Released
Thierry Carrez (ttx) on 2012-04-05
Changed in nova:
milestone: essex-4 → 2012.1
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers