Image remains in active state even image data is deleted from the rbd store

[Pranali Deore]

When glance is using rbd store, during image download, if delete call for the same image is attempted, delete fails with InUseByStore raised from the store but image data also gets deleted from the store and still image remains in active state.

Steps to reproduce:

1. Create image:
$ glance image-create --name test-new-image --file gentoo_root.img --disk-format raw --container-format bare

2. Download Image and simultaneously delete that image in other terminal:
$ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
[Errno 32] Corrupt image download. Hash was bc0b615f1a34f1bde273f496fa6246d33dbd760ac20e7687e55c1d06ab070907bfe087b2a25bceca3cd4ba819b096286d6e4f28a5f27398ed7afb0b0ba18721a expected e4f21997407b9cb0df347f6eba2feaeb14c19f15cf784da06b78e1d5ff776a419535c894dea10a859fa72bcb234e94ada0fc86de0ff127bf9280eede8d473edb

$ glance image-delete 1e1f9830-29e0-4901-8eff-ef05b7faf725
Unable to delete image '1e1f9830-29e0-4901-8eff-ef05b7faf725' because it is in use.

3. Try to Download the same Image again
$ glance image-download 1e1f9830-29e0-4901-8eff-ef05b7faf725 --file hhh.txt
Image 1e1f9830-29e0-4901-8eff-ef05b7faf725 has no data.

Log:

WARNING glance_store._drivers.rbd [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Remove image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 failed. It is in use.: rbd.ImageBusy: [errno 16] RBD image is busy (error removing image)
Dec 06 11:40:41 pdeore-bobdev <email address hidden>[813992]: WARNING glance.api.v2.images [None req-9d5aeaae-efc9-4d43-8496-2c742a52a4da admin admin] Image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 could not be deleted because it is in use: The image cannot be deleted because it is in use through the backend store outside of Glance.: glance_store.exceptions.InUseByStore: The image cannot be deleted because it is in use through the backend store outside of Glance.

Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: WARNING glance.image_cache.drivers.sqlite [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] ("Fetch of cache file failed (%(e)s), rolling back by moving '%(incomplete_path)s' to '%(invalid_path)s'", {'e': NotFound('RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist'), 'incomplete_path': '/opt/stack/data/glance/cache/incomplete/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4', 'invalid_path': '/opt/stack/data/glance/cache/invalid/c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4'}): glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache [None req-e71cfe15-f144-418b-b9f7-15afbece570b admin admin] Exception encountered while tee'ing image 'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' into cache: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist. Continuing with response.: glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache Traceback (most recent call last):
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 269, in __iter__
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache with rbd.Image(ioctx, self.name,
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache File "rbd.pyx", line 2894, in rbd.Image.__init__
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache rbd.ImageNotFound: [errno 2] RBD image not found (error opening image b'c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4' at snapshot b'snap')
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache During handling of the above exception, another exception occurred:
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache Traceback (most recent call last):
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/image_cache/__init__.py", line 352, in cache_tee_iter
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache for chunk in image_iter:
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache File "/opt/stack/glance/glance/notifier.py", line 412, in _get_chunk_data_iterator
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache for chunk in data:
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache File "/usr/local/lib/python3.10/dist-packages/glance_store/_drivers/rbd.py", line 282, in __iter__
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache raise exceptions.NotFound(
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache glance_store.exceptions.NotFound: RBD image c0fbdea3-5f53-40d0-a5f9-f9ade42dbbd4 does not exist
Dec 06 11:40:44 pdeore-bobdev <email address hidden>[813991]: ERROR glance.image_cache

[Pranali Deore (pranali-deore)]

There seems regression on Ceph side since when I tested with Ceph 'octopus' version, image gets deleted successfully without InUseByStore exception and download interrupts as usual saying hash is not correct.

This deletion issue is exists with Ceph 'reef', 'quincy' & 'pacific' versions.