3PAR: Snapshot stuck in Error_Deleting status

[Michael Denny]

3PAR Snapshot stuck in Error_Deleting status
Scenario
1. Create a volume vol1
2. Snapshot the volume vol1_snap
3. Create volume from snapshot vol1_snap_to_vol2
4. Try to delete snapshot

Talked to walt, we should be able to recover from this and return status to availalbe after sometime, instead of always being stuck in Error_Deleting status

2013-11-11 15:18:29.135 ERROR cinder.openstack.common.rpc.amqp [req-c8a80e4e-3679-42c6-919e-f82ab06a0970 0d4a18c85ff447d7b0dce7017793ce9c 0982d8dd597b40fb96b67e199137b06d] Exception during message handling
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp **args)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/utils.py", line 820, in wrapper
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp return func(self, *args, **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 425, in delete_snapshot
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp {'status': 'error_deleting'})
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/manager.py", line 413, in delete_snapshot
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp self.driver.delete_snapshot(snapshot_ref)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/openstack/common/lockutils.py", line 233, in inner
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp retval = f(*args, **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_fc.py", line 154, in delete_snapshot
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp self.common.delete_snapshot(snapshot)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/opt/stack/cinder/cinder/volume/drivers/san/hp/hp_3par_common.py", line 982, in delete_snapshot
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp self.client.deleteVolume(snap_name)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/hp3parclient/client.py", line 216, in deleteVolume
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp response, body = self.http.delete('/volumes/%s' % name)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 327, in delete
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp return self._cs_request(url, 'DELETE', **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 231, in _cs_request
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 205, in _time_request
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp resp, body = self.request(url, method, **kwargs)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/hp3parclient/http.py", line 199, in request
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp raise exceptions.from_response(resp, body)
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp HTTPConflict: Conflict (HTTP 409) 32 - volume has a child
2013-11-11 15:18:29.135 TRACE cinder.openstack.common.rpc.amqp

[Ramy Asselin (ramy-asselin)]

I reproduced the issue. Found a simple fix which returns this error in the log file and leaves the snapshot in the "available" state.

2013-12-18 17:20:16.705 ERROR cinder.volume.manager [req-d4386681-45cd-43a6-b0c8-83ba5838d870 9c55990936a440f6a4959b06eddf4673 02c63ce707ad43f49e2e6aed6967ba22] Cannot delete snapshot c0a18bc7-db8d-4092-8223-3d28859e2045: snapshot is busy

 989 except hpexceptions.HTTPConflict as ex:
 990 LOG.error(str(ex))
 991 raise exception.SnapshotIsBusy()

Will explore alternative fixes such as SunapshotInUse().

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65175

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65175
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=9834639454103ee2ba980628f0bc67ad0baa94ff
Submitter: Jenkins
Branch: master

commit 9834639454103ee2ba980628f0bc67ad0baa94ff
Author: Ramy Asselin <email address hidden>
Date: Mon Jan 6 15:26:07 2014 -0800

    3PAR: Raise Ex when del snapshot with depend vol

    Deleting a 3PAR snapshot can result in an HTTP Conflict error message,
    such as when a volume is dependent on it.
    Previously, this exception was not caught, and the state was set
    to "Error_Deleting" with no recovery path.
    The code was updated to catch this exception and raise a
    SnapShotIsBusy exception with an appropriate error message.
    The state reverts back to 'Available' allowing the user to try again
    after removing the dependency.

    Change-Id: Ibe8c5d581af10c85397c37993c49f8fc3bce8620
    Closes-Bug: #1250249