image data cannot be removed when deleting a saving status image

[Quan Guo]

The image data in /var/lib/glance/images/ cannot be removed when I delete a image that status is saving.

1. create a image
 glance image-create --name image-v1 --disk-format raw --container-format bare --file xx.image --is-public true

2. list the created image, the status is saving
[root@node2 ~]# glance image-list
+--------------------------------------+----------+-------------+------------------+------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+----------+-------------+------------------+------+--------+
| 00ec3d8d-41a5-4f7c-9448-694099a39bcf | image-v1 | raw | bare | 18 | saving |
+--------------------------------------+----------+-------------+------------------+------+--------+

3. delete the created image
glance image-delete image-v1

4. the image has been deleted but the image data still exists
[root@node2 ~]# glance image-list
+----+------+-------------+------------------+------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+----+------+-------------+------------------+------+--------+
+----+------+-------------+------------------+------+--------+

[root@node2 ~]# ls /var/lib/glance/images
00ec3d8d-41a5-4f7c-9448-694099a39bcf

This problem exists in both v1 and v2 API.

[yalei wang (yalei-wang)]

I think it is normal, should be cache mgt.

[Quan Guo (guoquan)]

Hi, ya lei, what do you mean about 'cache mgt'?

[Nikhil Komawar (nikhil-komawar)]

Let's see if someone gets bandwidth to confirm this soon.

[Jeremy Stanley (fungi)]

I remain dubious of the benefit of switching this bug to private after it was public (and for over a month at that).

[Zhi Yan Liu (lzy-dev)]

The key reason is that currently glance can receives new image location and save it to database only after its bits saving logic return (affect by glance and glance_store logics). So before upload execution 'thread' finish, glance doesn't know any location information of that 'saving' image, so delete 'thread' just think theres' no any locations/bits of the image need to remove. Figuring out solution.

[Stuart McLaren (stuart-mclaren)]

Thanks zhiyan.

Sounds like we need to save the location earlier, ie before we start writing the bits. Would that be possible?

[Zhi Yan Liu (lzy-dev)]

Hi Stuart,

Yes, it's a sensible solution, I'm thinking the details of this approach to make sure it's a best solution for this defect, it has some costs, e.g. how to get and handle these 'placeholder'/earlier location, and a little concerned on if this way can keep backward-compatible for glance code and client, e.g. not sure glance existing logics check image status instead of len(image.locations), outside as well, e.g. to make sure image under 'active' status before consume location data from metadata.

Thanks.

[Stuart McLaren (stuart-mclaren)]

Hmm, it's tricky because when the image is in the deleted state any access to the registry from the api using the user's context (ie token) will return a 404 for the user.

[Stuart McLaren (stuart-mclaren)]

Also, coordinating between an upload being processed on one api node and a delete coming in from another is tricky.

The upload won't be checking the database state so won't immediately detect that the image has been deleted.

I don't think we can disallow deleting images in a saving state either, because they could end up in a saving state eg if the upload node crashes.

It's almost like we need a form of opportunistic scrubbing even when delayed delete is not enabled.

[Zhi Yan Liu (lzy-dev)]

Fix proposed at https://review.openstack.org/#/c/144464/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/144464
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Submitter: Jenkins
Branch: master

commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/149387

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149397

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149387
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Submitter: Jenkins
Branch: stable/juno

commit 7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
     glance/api/v1/upload_utils.py
     glance/api/v2/image_data.py
     glance/tests/unit/test_domain_proxy.py
     glance/tests/unit/v1/test_api.py

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149646

[OpenStack Infra (hudson-openstack)]

Change abandoned by nikhil komawar (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/149397
Reason: https://review.openstack.org/#/c/149646

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/149646
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=f1260cc771ee068651aa62b972bef49d9af81eb0
Submitter: Jenkins
Branch: stable/icehouse

commit f1260cc771ee068651aa62b972bef49d9af81eb0
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.
    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.
    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
            glance/api/v1/upload_utils.py
            glance/api/v2/image_data.py
            glance/quota/__init__.py
            glance/tests/unit/test_domain_proxy.py
            glance/tests/unit/test_quota.py
            glance/tests/unit/v1/test_api.py

    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60