glusterFS_native_driver: snapshot delete doesn't delete snapshot entries that are in error state

[karthick]

Description:

In openstack-manila running kilo release, if snapshot creation fails for some reason, an entry for snapshot is still created with 'error' state. Attempting to delete this invalid entry of snapshot doesn't succeed.

Possible scenarios which could create a snapshot in error state are when,

 - gluster daemon is down in the backend gluster node
 - gluster node is down
 - gluster brick process is down
 - gluster volume is down
 - snapshot created by manila is deleted from backend gluster

While it is okay to create a snapshot with error state, attempting to delete or force-delete should clear the entry from database. Administrator has no means to remove these snapshots in error state.

Error msg from the logs:

GlusterfsException: Failed to identify backing GlusterFS object for snapshot 0a6e89a3-cbc3-4322-bcb7-05282b3b9816 of share 78fea54d-78f6-4d18-b478-4b40ad2b7c30: a single candidate was expected, 0 was found.

CLI output:

# manila snapshot-list
+--------------------------------------+--------------------------------------+--------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+--------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error | snap-01 | 1 |
+--------------------------------------+--------------------------------------+--------+---------+------------+
# manila snapshot-delete fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error_deleting | snap-01 | 1 |
+--------------------------------------+--------------------------------------+----------------+---------+------------+

# manila snapshot-reset-state fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+-----------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+-----------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | available | snap-01 | 1 |
+--------------------------------------+--------------------------------------+-----------+---------+------------+

# manila snapshot-force-delete fac9835a-9689-46a0-8ee7-d2f2d12ad486
# manila snapshot-list
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| ID | Share ID | Status | Name | Share Size |
+--------------------------------------+--------------------------------------+----------------+---------+------------+
| fac9835a-9689-46a0-8ee7-d2f2d12ad486 | 68c37e73-9710-430a-8f39-7d79ad6af669 | error_deleting | snap-01 | 1 |
+--------------------------------------+--------------------------------------+----------------+---------+------------+

Steps to reproduce:

1) Try to create a snapshot when volume is down (one of the possible ways to create a snapshot that would end up in error state)
2) A snapshot in error state is created
3) Attempt to delete or force-delete the snapshot - Snapshot doesn't gets deleted

[Ben Swartzlander (bswartz)]

We discussed this in the channel and it's actually a behavior of the manager not the driver. We're open to considering changing the behavior of the manager, but not as a bugfix for Liberty -- it would a change for Mitaka.

[Dustin Schoenbrun (dschoenb)]

The GlusterFS driver is no longer actively maintained.