volume created from volume with RBDDriver delete failed if deferred deletion is enabled

[Yang Zhang]

Now cinder has supported deferred deletion in RBD, when we enabled this feature deleting volumes created from volume will be failed.
I checked the code: deleting a volume created from volume will remove the volume's clone snap immediately after the volume is removed or the volume is moved to trash. But we cannot unprotect the volumes'clone snap beforce this volume is purged in rbd trash. In my test ceph version 12.2.11 will return imageNotExist Excepthion, and ceph 13.2.1 will return imageBusy Exception。 By the way, if we force delete the volume later after becoming error_deleting, the volume can be deleted successfully because the rbd image does not exist any more. but its parent clone snap will be left, and cannot be deleted with cinder.
And similar Error will occur if we create a volume from a cinder snapshot, then delete the volume, then we delete the snapshot while the volume is still in rbd trash.
I've no idea how we should deal with this scenario. I added a table in cinder db to save these snaps and a periodic job to try to delete them later. Hoping to find other better way.

[Yang Zhang (wosunoozzy)]

Some log deleting snapshot with ceph 12.2.10:
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server [req-732fd601-576e-4a22-ae0e-35f1b6e62b62 fc26707fe5254e729431d3017250949e d900ebe92343410888fcd10c6c7737fb - - -] Exception during message handling: ImageNotFound: [errno 2] error listing images
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "<string>", line 2, in delete_snapshot
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server return f(*a, **k)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1205, in delete_snapshot
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server snapshot.save()
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in _exit_
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1195, in delete_snapshot
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server self.driver.delete_snapshot(snapshot)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 1335, in delete_snapshot
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server children_list = self._get_children_info(volume, snap_name)
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py", line 1156, in _get_children_info
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server children_list = volume.list_children()
2019-04-11 15:00:00.720 100073 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/eventle...

[Jing Geng (jinggeng)]

Hi @jbernard, are you still workinng on this? If not, please un-assign yourself from this bug since I plan to create a patch

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/cinder/+/900111