image download process won't terminate after deleting image from dashboard

[Zhong Chaoliang]

Version: Grizzly on ubuntu 12.04

Firstly, create an image with the dashboard, and input the url of the image, rather than selecting a local image file.
Once the status of the image becomes 'saving', delete it from the dashboard.
The image is removed from the dashboard as expected, however, the downloading process continues in the background.
If you have a look at /var/lib/glance/images/ , you'll find the size of the image is growing.

Sorry, I'm not sure whether it is an issue of glance or horizon.

[Victoria Martinez de la Cruz (vkmc)]

I could reproduce this in a DevStack deployment on Ubuntu 12.04 LTS.

## Glance images folder

vkmc@havana:/opt/stack/data/glance/images$ ls -la
total 33052
drwxrwxr-x 2 vkmc vkmc 4096 Jul 3 12:22 .
drwxrwxr-x 4 vkmc vkmc 4096 Jul 3 12:09 ..
-rw-r----- 1 vkmc vkmc 0 Jul 3 12:22 1aa74a9a-67d0-4005-b78e-0936be8770aa <-
-rw-r----- 1 vkmc vkmc 4955792 Jul 3 12:11 2fb167b6-b0c4-4d78-8667-cefd077433a6
-rw-r----- 1 vkmc vkmc 25165824 Jul 3 12:12 b0b77bb0-36c5-465b-a102-9e282e908b8f
-rw-r----- 1 vkmc vkmc 3714968 Jul 3 12:11 be71ae20-15c5-4dd0-b1d8-44de8afe044d
vkmc@havana:/opt/stack/data/glance/images$ ls -la
total 42000
drwxrwxr-x 2 vkmc vkmc 4096 Jul 3 12:22 .
drwxrwxr-x 4 vkmc vkmc 4096 Jul 3 12:09 ..
-rw-r----- 1 vkmc vkmc 9159168 Jul 3 12:22 1aa74a9a-67d0-4005-b78e-0936be8770aa <-
-rw-r----- 1 vkmc vkmc 4955792 Jul 3 12:11 2fb167b6-b0c4-4d78-8667-cefd077433a6
-rw-r----- 1 vkmc vkmc 25165824 Jul 3 12:12 b0b77bb0-36c5-465b-a102-9e282e908b8f
-rw-r----- 1 vkmc vkmc 3714968 Jul 3 12:11 be71ae20-15c5-4dd0-b1d8-44de8afe044d

## image_locations table after bug reproduction

It's present in the Glance image_locations table

mysql> select * from image_locations;
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
| id | image_id | value | created_at | updated_at | deleted_at | deleted |
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
| 1 | 2fb167b6-b0c4-4d78-8667-cefd077433a6 | file:///opt/stack/data/glance/images/2fb167b6-b0c4-4d78-8667-cefd077433a6 | 2013-07-03 15:11:58 | 2013-07-03 15:11:58 | NULL | 0 |
| 2 | be71ae20-15c5-4dd0-b1d8-44de8afe044d | file:///opt/stack/data/glance/images/be71ae20-15c5-4dd0-b1d8-44de8afe044d | 2013-07-03 15:11:59 | 2013-07-03 15:11:59 | NULL | 0 |
| 3 | b0b77bb0-36c5-465b-a102-9e282e908b8f | file:///opt/stack/data/glance/images/b0b77bb0-36c5-465b-a102-9e282e908b8f | 2013-07-03 15:12:01 | 2013-07-03 15:12:01 | NULL | 0 |
| 4 | 1aa74a9a-67d0-4005-b78e-0936be8770aa | file:///opt/stack/data/glance/images/1aa74a9a-67d0-4005-b78e-0936be8770aa | 2013-07-03 15:22:39 | 2013-07-03 15:22:39 | NULL | 0 | <-
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
4 rows in set (0.00 sec)

## image-list after bug reproduction

But, it's not in the image-list

vkmc@havana:/opt/stack/data/glance/images$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+--------------------...

[Gabriel Hurley (gabriel-hurley)]

From the description I'm 95% certain this is an issue on the glance end of things. If I'm wrong, please re-open it for Horizon.

[Feilong Wang (flwang)]

I can create this issue based on latest devstack. And after reviewed the database record, there is no image created yet. So seems after cancel the image creation, there is no good way to tell glance to remove the image data.

[Feilong Wang (flwang)]

typo. I mean I can recreate it.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/121639

[Nikhil Komawar (nikhil-komawar)]

Glance side update:

(To be) abandoned: https://review.openstack.org/121639

Complete fix proposed: https://review.openstack.org/#/c/144464/

https://review.openstack.org/#/c/144464/ resolves the issue on Glance side. However, the download process may still be running from dashboard in some cases.

We've a race condition between when the download has started and when the upload operation raises a Duplicate exception.

One suggestion is to poll for Image status and delete the download process on the client (here Dashboard) side.

[OpenStack Infra (hudson-openstack)]

Change abandoned by Mike Fedosin (<email address hidden>) on branch: master
Review: https://review.openstack.org/121639

[Tristan Cacqueray (tristan-cacqueray)]

Can someone please "nominate" this fix for Juno and Icehouse ?

And the backport is not trivial, tests seems to require a tiny bit of rework, can someone also submit https://review.openstack.org/#/c/144464/ for stable/Juno and stable/Icehouse branch ?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144464
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Submitter: Jenkins
Branch: master

commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>

[Ian Cordasco (icordasc)]

Tristan, this has been nominated. If someone doesn't beat me, I'll take a crack at backporting this.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/149387

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149387
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Submitter: Jenkins
Branch: stable/juno

commit 7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
     glance/api/v1/upload_utils.py
     glance/api/v2/image_data.py
     glance/tests/unit/test_domain_proxy.py
     glance/tests/unit/v1/test_api.py

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149646

[OpenStack Infra (hudson-openstack)]

Change abandoned by nikhil komawar (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/149397
Reason: https://review.openstack.org/#/c/149646

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149646
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=f1260cc771ee068651aa62b972bef49d9af81eb0
Submitter: Jenkins
Branch: stable/icehouse

commit f1260cc771ee068651aa62b972bef49d9af81eb0
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.
    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.
    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
            glance/api/v1/upload_utils.py
            glance/api/v2/image_data.py
            glance/quota/__init__.py
            glance/tests/unit/test_domain_proxy.py
            glance/tests/unit/test_quota.py
            glance/tests/unit/v1/test_api.py

    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60