delete gluster volume snapshot is stuck in deleting forever after storage issue

[Dafna Ron]

we had an outage and after the server came up we still had problems with the storage which I did not know about.
I send several commands to delete snapshots and to delete volumes.
after fixing the problem with the storage, the delete of the volume completed but the snapshots are stuck in deleting

[root@cougar06 ~(keystone_admin)]# cinder snapshot-list
+--------------------------------------+--------------------------------------+----------+--------------+------+
| ID | Volume ID | Status | Display Name | Size |
+--------------------------------------+--------------------------------------+----------+--------------+------+
| 4a6597fc-d33c-49b0-88e4-3dc0a6e9855c | 1560fa00-752b-4d7b-a747-3ef9bf483692 | deleting | None | 1 |
| 4ac236e1-61f1-47dc-a0fa-d96162622121 | 1560fa00-752b-4d7b-a747-3ef9bf483692 | deleting | None | 1 |
+--------------------------------------+--------------------------------------+----------+--------------+------+

Steps to Reproduce:
1. configure cinder to work with gluster
2. create a volume and a snapshot
3. shut down gluster
4. send a delete snapshot command
5. restart gluster and mount

[Avishay Traeger (avishay-il)]

Can you please submit logs to help track down the issue?

[Dafna Ron (dron-3)]

attached

[John Griffith (john-griffith)]

So the first problem is that we shouldn't allow you to delete a volume while it still has a snapshot, we'll need to figure out how you got past the api/manager checks here but in the meantime.

Have you looked at just using the cinder client reset-state cmd? You can try applying reset-state to available on the snapshot and then issue the delete cmd again. This is assuming you've succesfully deleted it from the gluster back-end.

Let me know and we'll go from there.

[Thang Pham (thang-pham)]

I tried to reproduce this bug by following the steps above. After I stopped the gluster volume that I used to hold my cinder volumes and tried to delete the cinder volume snapshot, I got an error_deleting, which was expected. The actual error in the log says:

Returning exception GlusterFS share at /opt/stack/data/cinder/mnt/acbc597177d8b8303be1e35369494017 is not writable by the Cinder volume service. Snapshot operations will not be supported. to caller

Once I restarted the gluster volume and issued a cinder snapshot-reset-state on the snapshot, I was able to successfully delete it.

Are you still encountering this problem in the latest Icehouse release?