CEPH RBD: Snapshot stuck in Error_Deleting status

[Rey Aram A. Alcantara]

I have set the cinder config to debug enabled but there are no debug or error logs and it returns a success delete but if you check the list it will be stuck to an error_deleting status you can see it on the file that I attached.

[Rey Aram A. Alcantara (reyaram-alcantara)]

I have tried to debug and here is the log when I deleted the volume snapshot:

2014-08-27 20:05:50.821 26525 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'fba8a3983484473ebb5caf6af441c282', 'tenant': u'88f36feb0cc6464f83bdfb20ba7b9f82', 'user_identity': u'fba8a3983484473ebb5caf6af441c282 88f36feb0cc6464f83bdfb20ba7b9f82 - - -'}
2014-08-27 20:05:50.848 26525 INFO cinder.volume.manager [req-6c3decaf-9a4b-4e98-87b9-03dc2c167d0b fba8a3983484473ebb5caf6af441c282 88f36feb0cc6464f83bdfb20ba7b9f82 - - -] snapshot dc883787-3b29-49c9-a377-117a2d0f2f58: deleting
2014-08-27 20:05:50.975 26525 ERROR oslo.messaging.rpc.dispatcher [req-6c3decaf-9a4b-4e98-87b9-03dc2c167d0b fba8a3983484473ebb5caf6af441c282 88f36feb0cc6464f83bdfb20ba7b9f82 - - -] Exception during message handling: error removing snapshot snapshot-dc883787-3b29-49c9-a377-117a2d0f2f58 from volume-6506c99a-40ff-4877-8e6f-43324c7a0827
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 166, in lso_inner1
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher return lso_inner2(inst, context, snapshot_id, **kwargs)
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/lockutils.py", line 233, in inner
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher retval = f(*args, **kwargs)
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 165, in lso_inner2
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher return f(*_args, **_kwargs)
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/volume/manager.py", line 542, in delete_snapshot
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher {'status': 'error_deleting'})
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/cinder/openstack/common/excutils.py", line 68, in __exit__
2014-08-27 20:05:50.975 26525 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, se...

[Edward Hope-Morley (hopem)]

Rey, if volumes have been created from these snapshots (and they still exist) and config.rbd_flatten_volume_from_snapshot is False (default) then you would expect to see this see this error. Only when volumes created from these snapshots are deleted can the snapshots themselves be deleted. Can you confirm whether this is the case for you?

[Rey Aram A. Alcantara (reyaram-alcantara)]

Upon checking this would be the case, I have not set rbd_flatten_volume_from_snapshot to my cinder config file.

I will update my config file, then test again.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/151257

[Sean McGinnis (sean-mcginnis)]

Automatically unassigning due to inactivity.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/151257
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c776607c4305304f56e779403e1d23ad51e8dcb5
Submitter: Jenkins
Branch: master

commit c776607c4305304f56e779403e1d23ad51e8dcb5
Author: Dirk Mueller <email address hidden>
Date: Thu Jan 29 14:11:35 2015 +0100

    RBD: Make snapshot_delete more robust

    Since there is no equivalent of a force-snapshot-delete,
    the normal snapshot-delete should be more robust to
    also handle backend errors. In case the backend does not have
    the image snapshot anymore, log an info message and succeed the
    operation.

    Change-Id: I7fe0878dbc07053ac78272b6998513fafa1c36e8
    Closes-Bug: #1415905
    Related-Bug: #1361926

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/299989

[OpenStack Infra (hudson-openstack)]

Change abandoned by Saverio Proto (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/299989
Reason: I will try this again in Mitaka when it is merged in master