| 2014-02-04 13:02:37 |
Abhishek Kekane |
bug |
|
|
added bug |
| 2014-02-04 13:02:37 |
Abhishek Kekane |
attachment added |
|
steps-to-reproduce.txt https://bugs.launchpad.net/bugs/1276142/+attachment/3968664/+files/steps-to-reproduce.txt |
|
| 2014-02-04 13:03:08 |
Abhishek Kekane |
glance: assignee |
|
Abhishek Kekane (abhishek-kekane) |
|
| 2014-02-04 17:52:43 |
Abhishek Kekane |
attachment removed |
steps-to-reproduce.txt https://bugs.launchpad.net/glance/+bug/1276142/+attachment/3968664/+files/steps-to-reproduce.txt |
|
|
| 2014-02-04 17:53:53 |
Abhishek Kekane |
description |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
Please refer steps-to-reproduce.txt to reproduce the issue.
Possible solution:
Image status should not get updated in database to "deleted" if response is HTTPForbidden (HTTP 403) or other than HTTP 200 |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
Possible solution:
Image status should not get updated in database to "deleted" if response is HTTPForbidden (HTTP 403) or other than HTTP 200 |
|
| 2014-02-05 06:22:32 |
Abhishek Kekane |
description |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
Possible solution:
Image status should not get updated in database to "deleted" if response is HTTPForbidden (HTTP 403) or other than HTTP 200 |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
In the longevity test we have encoutered this issue, IMO there is some issue with OS and it is throwing "Forbidden" exception and somehow its not deleting that file, so inorder to reproduce this issue please follow the below mentioned steps:
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
This issue is reproducible only in v1 version. In case of v2 version, it doesn't delete the meta data associated with the image before actually deleting it from the backend store.
In glance, if you enable delayed_delete = True and run glance-scrubber service, glance api will put the image in the queue when the image is deleted by the user and this image will be deleted asynchronously. If it encounters the above reported issue while deleting the image in the glance-scrubber, then this service will simply log error message and keep on retrying deleting the image until it is deleted finally.
Possible solution:
1. When delayed_delete is enabled.
delete_image_metadata method should be moved to the scrubber and called immediately after setting the image status from "pending_delete" to "deleted".
2. When delayed_delete is not enabled.
delete_image_metadata method should be called after deleting the actual image from the backend.
i.e. after upload_utils.initiate_deletion(req, image['location'], id, CONF.delayed_delete) is called
Also the status of the image should be set back to the previous state in the forbidden exception block. |
|
| 2014-02-05 06:23:21 |
Abhishek Kekane |
description |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
In the longevity test we have encoutered this issue, IMO there is some issue with OS and it is throwing "Forbidden" exception and somehow its not deleting that file, so inorder to reproduce this issue please follow the below mentioned steps:
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
This issue is reproducible only in v1 version. In case of v2 version, it doesn't delete the meta data associated with the image before actually deleting it from the backend store.
In glance, if you enable delayed_delete = True and run glance-scrubber service, glance api will put the image in the queue when the image is deleted by the user and this image will be deleted asynchronously. If it encounters the above reported issue while deleting the image in the glance-scrubber, then this service will simply log error message and keep on retrying deleting the image until it is deleted finally.
Possible solution:
1. When delayed_delete is enabled.
delete_image_metadata method should be moved to the scrubber and called immediately after setting the image status from "pending_delete" to "deleted".
2. When delayed_delete is not enabled.
delete_image_metadata method should be called after deleting the actual image from the backend.
i.e. after upload_utils.initiate_deletion(req, image['location'], id, CONF.delayed_delete) is called
Also the status of the image should be set back to the previous state in the forbidden exception block. |
If the deletion of physical image failed with OSError for some reason, it is not possible to delete the image using image-delete api call.
In the longevity test we have encountered this issue, IMO there is some issue with OS and it is throwing "Forbidden" exception and somehow its not deleting that file, so in order to reproduce this issue please follow the below mentioned steps:
Steps to reproduce:
1. image creation, upload
openstack@opencloud1:~$ glance image-create
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | None |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | None |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 0 |
| status | queued |
| updated_at | 2014-01-29T02:27:07 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ glance image-update 675a82ab-4515-451a-932b-6da7f8bce056 --file images/CorePlus4.7.1.qcow2 --disk-format qcow2 --container-format bare
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | False |
| deleted_at | None |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| name | None |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | active |
| updated_at | 2014-01-29T02:27:20 |
+------------------+--------------------------------------+
2. write protect the image file
openstack@opencloud1:~$ sudo chattr +i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
3. delete the image using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
4. Image is logically deleted, but physically remaining
openstack@opencloud1:~$ glance image-show 675a82ab-4515-451a-932b-6da7f8bce056
+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | a5282e9259f822c782bc7aea8a8870c6 |
| container_format | bare |
| created_at | 2014-01-29T02:27:07 |
| deleted | True |
| deleted_at | 2014-01-29T02:30:49 |
| disk_format | qcow2 |
| id | 675a82ab-4515-451a-932b-6da7f8bce056 |
| is_public | False |
| min_disk | 0 |
| min_ram | 0 |
| owner | de4a6631051a4df09bc1b12923244296 |
| protected | False |
| size | 43450368 |
| status | deleted |
| updated_at | 2014-01-29T02:30:49 |
+------------------+--------------------------------------+
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
5. Remove the write-protect from image file
openstack@opencloud1:~$ sudo chattr -i /opt/stack/data/glance/images/675a82ab-4515-451a-932b-6da7f8bce056
6. Try to delete the image again using "glance image-delete"
openstack@opencloud1:~$ glance image-delete 675a82ab-4515-451a-932b-6da7f8bce056
Request returned failure status.
HTTPForbidden (HTTP 403)
7. Image file will not get deleted
openstack@opencloud1:~$ sudo ls -ls /opt/stack/data/glance/images/
total 33088
42432 -rw-r----- 1 glance glance 43450368 Jan 29 11:27 675a82ab-4515-451a-932b-6da7f8bce056
This issue is reproducible only in v1 version. In case of v2 version, it doesn't delete the meta data associated with the image before actually deleting it from the backend store.
In glance, if you enable delayed_delete = True and run glance-scrubber service, glance api will put the image in the queue when the image is deleted by the user and this image will be deleted asynchronously. If it encounters the above reported issue while deleting the image in the glance-scrubber, then this service will simply log error message and keep on retrying deleting the image until it is deleted finally.
Possible solution:
1. When delayed_delete is enabled.
delete_image_metadata method should be moved to the scrubber and called immediately after setting the image status from "pending_delete" to "deleted".
2. When delayed_delete is not enabled.
delete_image_metadata method should be called after deleting the actual image from the backend.
i.e. after upload_utils.initiate_deletion(req, image['location'], id, CONF.delayed_delete) is called
Also the status of the image should be set back to the previous state in the forbidden exception block. |
|
| 2014-02-07 04:52:03 |
Abhishek Kekane |
summary |
Physical image can not be deleted again if deletion is failed with OSError for some reason |
Physical image can not be deleted again if deletion is failed with OSError |
|
| 2014-02-07 14:34:03 |
OpenStack Infra |
glance: status |
New |
In Progress |
|
| 2014-02-08 05:56:28 |
Zhi Yan Liu |
glance: importance |
Undecided |
Medium |
|
| 2014-02-08 08:56:33 |
Tomoko Inoue |
bug |
|
|
added subscriber Tomoko Inoue |
| 2014-02-20 10:09:46 |
OpenStack Infra |
glance: status |
In Progress |
Fix Committed |
|
| 2014-03-05 10:24:13 |
Thierry Carrez |
glance: status |
Fix Committed |
Fix Released |
|
| 2014-03-05 10:24:13 |
Thierry Carrez |
glance: milestone |
|
icehouse-3 |
|
| 2014-04-17 11:12:22 |
Thierry Carrez |
glance: milestone |
icehouse-3 |
2014.1 |
|
