Delete image call fails with 500 during image download is in progress

Bug #2070258 reported by Pranali Deore
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
In Progress
High
Pranali Deore

Bug Description

When glance is using cinder backend, and during image download if delete call for the same image has been attempted, delete call fails 'HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.': Unable to delete image 'cf51102f-89a1-481d-adfc-2aa02b641d02' which should give 409 HTTPConflict error to the user.

Steps to reproduce:

1. Create Image
$ glance image-create-via-import --container-format bare --disk-format qcow2 --name test-img --import-method web-download --uri https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-aarch64-disk.img --store lvmdriver-1

2. Download the Image once it is active and simultaneously delete the image in other terminal
$ glance image-download 6a63cfe6-c10c-4319-8072-44db1012c6c3 --file test.txt

$ glance image-delete 6a63cfe6-c10c-4319-8072-44db1012c6c3
'HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.': Unable to delete image '6a63cfe6-c10c-4319-8072-44db1012c6c3'

g-api Log:

Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi [None req-78ab744d-169d-44c2-ae6a-fcd74caf2e4a admin admin] Caught error: Failed to delete volume b54a9b0f-758d-481a-904e-a450edc73c50: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-68dd17d7-5013-40f9-8052-06f195b8033f): glance_store.exceptions.BackendException: Failed to delete volume b54a9b0f-758d-481a-904e-a450edc73c50: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-68dd17d7-5013-40f9-8052-06f195b8033f)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi Traceback (most recent call last):
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/cinder/store.py", line 1123, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi self.volume_api.delete(client, loc.volume_id)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/glance_store/common/cinder_utils.py", line 71, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi client.volumes.delete(volume_id)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/cinderclient/v3/volumes_base.py", line 235, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return self._delete(loc)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/cinderclient/base.py", line 313, in _delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi resp, body = self.api.client.delete(url)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/cinderclient/client.py", line 229, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return self._cs_request(url, 'DELETE', **kwargs)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/cinderclient/client.py", line 211, in _cs_request
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return self.request(url, method, **kwargs)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/cinderclient/client.py", line 197, in request
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi raise exceptions.from_response(resp, body)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi cinderclient.exceptions.BadRequest: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-68dd17d7-5013-40f9-8052-06f195b8033f)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi During handling of the above exception, another exception occurred:
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi Traceback (most recent call last):
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1302, in __call__
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action,
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/wsgi.py", line 1345, in dispatch
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return method(*args, **kwargs)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/utils.py", line 415, in wrapped
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return func(self, req, *args, **kwargs)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/api/v2/images.py", line 906, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi image.delete()
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 200, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi self.base.delete()
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/domain/proxy.py", line 200, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi self.base.delete()
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/location.py", line 458, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi self.store_utils.delete_image_location_from_backend(
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/store_utils.py", line 130, in delete_image_location_from_backend
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi safe_delete_from_backend(context, image_id, location)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/opt/stack/glance/glance/common/store_utils.py", line 57, in safe_delete_from_backend
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi ret = store_api.delete(location['url'],
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/glance_store/multi_backend.py", line 510, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return store.delete(loc, context=context)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/glance_store/capabilities.py", line 176, in op_checker
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/cinder/store.py", line 1129, in delete
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi raise exceptions.BackendException(msg)
Jun 24 10:35:02 pranali-new-devstack <email address hidden>[2485343]: ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to delete volume b54a9b0f-758d-481a-904e-a450edc73c50: Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots, awaiting a transfer, or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: req-68dd17d7-5013-40f9-8052-06f195b8033f)

Changed in glance:
assignee: nobody → Pranali Deore (pranali-deore)
importance: Undecided → High
Revision history for this message
Abhishek Kekane (abhishek-kekane) wrote :

What happens to download, is it completed successfully?

Also I would like to know if download also interrupts and image is still in active will it gets download completely on second download call?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/glance/+/922623

Changed in glance:
status: New → In Progress
affects: glance → glance-store
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.