Physical image can not be deleted again if deletion is failed with OSError

[Abhishek Kekane]

If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.

In the longevity test we have encountered this issue, IMO there is some issue with OS and it is throwing "Forbidden" exception and somehow its not deleting that file, so in order to reproduce this issue please follow the below mentioned steps:

Steps to reproduce:

1. image creation, upload

    openstack@opencloud1:~$ glance image-create
    +------------------+--------------------------------------+
    | Property | Value |
    +------------------+--------------------------------------+
    | checksum | None |
    | container_format | None |
    | created_at | 2014-01-29T02:27:07 |
    | deleted | False |
    | deleted_at | None |
    | disk_format | None |
    | id | 675a82ab-4515-451a-932b-6da7f8bce056 |
    | is_public | False |
    | min_disk | 0 |
    | min_ram | 0 |
    | name | None |
    | owner | de4a6631051a4df09bc1b12923244296 |
    | protected | False |
    | size | 0 |
    | status | queued |
    | updated_at | 2014-01-29T02:27:07 |
    +------------------+--------------------------------------+

    openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
    +------------------+--------------------------------------+
    | Property | Value |
    +------------------+--------------------------------------+
    | checksum | a5282e9259f822c782bc7aea8a8870c6 |
    | container_format | bare |
    | created_at | 2014-01-29T02:27:07 |
    | deleted | False |
    | deleted_at | None |
    | disk_format | qcow2 |
    | id | 675a82ab-4515-451a-932b-6da7f8bce056 |
    | is_public | False |
    | min_disk | 0 |
    | min_ram | 0 |
    | name | None |
    | owner | de4a6631051a4df09bc1b12923244296 |
    | protected | False |
    | size | 43450368 |
    | status | active |
    | updated_at | 2014-01-29T02:27:20 |
    +------------------+--------------------------------------+

2. write protect the image file

    openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056

3. delete the image using "glance image-delete"

    openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
    Request returned failure status.
    HTTPForbidden (HTTP 403)

4. Image is logically deleted, but physically remaining

    openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056

    +------------------+--------------------------------------+
    | Property | Value |
    +------------------+--------------------------------------+
    | checksum | a5282e9259f822c782bc7aea8a8870c6 |
    | container_format | bare |
    | created_at | 2014-01-29T02:27:07 |
    | deleted | True |
    | deleted_at | 2014-01-29T02:30:49 |
    | disk_format | qcow2 |
    | id | 675a82ab-4515-451a-932b-6da7f8bce056 |
    | is_public | False |
    | min_disk | 0 |
    | min_ram | 0 |
    | owner | de4a6631051a4df09bc1b12923244296 |
    | protected | False |
    | size | 43450368 |
    | status | deleted |
    | updated_at | 2014-01-29T02:30:49 |
    +------------------+--------------------------------------+

    openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/

    total 33088
    42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056

5. Remove the write-protect from image file

    openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056

6. Try to delete the image again using "glance image-delete"

    openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
    Request returned failure status.
    HTTPForbidden (HTTP 403)

7. Image file will not get deleted

    openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/

    total 33088
    42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056

This issue is reproducible only in v1 version. In case of v2 version, it doesn't delete the meta data associated with the image before actually deleting it from the backend store.

In glance, if you enable delayed_delete = True and run glance-scrubber service, glance api will put the image in the queue when the image is deleted by the user and this image will be deleted asynchronously. If it encounters the above reported issue while deleting the image in the glance-scrubber, then this service will simply log error message and keep on retrying deleting the image until it is deleted finally.

Possible solution:

1. When delayed_delete is enabled.
delete_image_metadata method should be moved to the scrubber and called immediately after setting the image status from "pending_delete" to "deleted".

2. When delayed_delete is not enabled.
delete_image_metadata method should be called after deleting the actual image from the backend.
i.e. after upload_utils.initiate_deletion(req, image['location'], id, CONF.delayed_delete) is called
Also the status of the image should be set back to the previous state in the forbidden exception block.

[Feilong Wang (flwang)]

Why use want to touch the image repository manually in step 2? Is there any specific requirement?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/71913

[Abhishek Kekane (abhishek-kekane)]

Hi Fei Long Wang,

Basically while running longevity test we have encountered this issue.
IMO there is some issue with OS and it is throwing "Forbidden" exception and somehow its not deleting that file.
Its hard to reproduce the issue, so I have tried using creating the image file write protected manually.

[Mark Washenberger (markwash)]

It seems like this is going a bit far for a problem with the underlying system.

I think we should still delete the image (metadata) but just log appropriately so that operators can clean up their disk.

[Zhi Yan Liu (lzy-dev)]

@markwash, for this issue IMO make v1 behavior be consistent with v2 is make more sense, so do you think to do some changes in v1controller is a good way to go? And even underneath storage meet some issues but IMHO it's a worth thng if we can keep the consistent of image metadata/image status in glance layer.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/71913
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=f50074c0f65fdeae90dcd4b469fa2ac6f306e4b7
Submitter: Jenkins
Branch: master

commit f50074c0f65fdeae90dcd4b469fa2ac6f306e4b7
Author: abhishek-kekane <email address hidden>
Date: Wed Feb 5 05:33:13 2014 -0800

    Delete image metadata after image is deleted

    Delete image metadata after the image is deleted successfully from
    the backend store.
    Set image status to it's original status if the image deletion fails.

    Change-Id: I5fee1bace58ff49a8952292250629c1628edbac8
    Closes-Bug: #1276142