Heat stack deletion followed by an unsuccessful heat stack delete attempt resulted in calling begin_detaching on a volume which was already in detaching state. The first heat stack delete attempt failed due to a non-responsive nova-compute. Nova-api called begin_detaching which was successful, but detach_volume didn't run in nova-compute and hence the volume remained in 'detaching'. During the second attempt, begin_detaching returned success instead of failing the operation.
2015-06-10 14:00:55.084 8559 DEBUG routes.middleware [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Matched GET /df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7 __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2015-06-10 14:00:55.085 8559 DEBUG routes.middleware [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Route path: '/{project_id}/volumes/:(id)', defaults: {'action': u'show', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7faef9da60d0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2015-06-10 14:00:55.085 8559 DEBUG routes.middleware [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Match dict: {'action': u'show', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7faef9da60d0>, 'project_id': u'df97441007334e15a7e45e6564e6fee0', 'id': u'7f0ca54c-b481-4ab6-8534-969ceb3062e7'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2015-06-10 14:00:55.086 8559 INFO cinder.api.openstack.wsgi [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] GET http://10.155.21.150:8776/v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7
2015-06-10 14:00:55.086 8559 DEBUG cinder.api.openstack.wsgi [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Empty body provided in request get_body /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:787
2015-06-10 14:00:55.105 8559 AUDIT cinder.api.v1.volumes [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] vol={'migration_status': None, 'availability_zone': u'nova', 'terminated_at': None, 'updated_at': datetime.datetime(2015, 6, 10, 5, 1, 17), 'provider_geometry': None, 'snapshot_id': None, 'ec2_id': None, 'mountpoint': u'/dev/sdb', 'deleted_at': None, 'id': u'7f0ca54c-b481-4ab6-8534-969ceb3062e7', 'size': 10L, 'user_id': u'a8e3e0c5875445faae2c10944600e77a', 'attach_time': u'2015-06-10T04:41:42.102049', 'attached_host': None, 'display_description': None, 'volume_admin_metadata': [<cinder.db.sqlalchemy.models.VolumeAdminMetadata object at 0x7faef89dfd50>, <cinder.db.sqlalchemy.models.VolumeAdminMetadata object at 0x7faef89df3d0>], 'encryption_key_id': None, 'project_id': u'df97441007334e15a7e45e6564e6fee0', 'launched_at': datetime.datetime(2015, 6, 10, 4, 8, 20), 'scheduled_at': datetime.datetime(2015, 6, 10, 4, 8, 20), 'status': u'detaching', 'volume_type_id': None, 'deleted': False, 'provider_location': None, 'host': u'controller02', 'source_volid': None, 'provider_auth': None, 'display_name': u'stack_500_3-my_vol111-j5bqzus6ozyc', 'instance_uuid': u'76e4718a-a474-44ae-8f77-b0016326808f', 'bootable': False, 'created_at': datetime.datetime(2015, 6, 10, 4, 8, 20), 'attach_status': u'attached', 'volume_type': None, '_name_id': None, 'volume_metadata': [], 'metadata': {u'readonly': u'False', u'attached_mode': u'rw'}}
2015-06-10 14:00:55.160 8559 INFO cinder.api.openstack.wsgi [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] http://10.155.21.150:8776/v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7 returned with HTTP 200
2015-06-10 14:00:55.161 8559 INFO eventlet.wsgi.server [req-55a01c23-6685-4b11-a560-e9ac146caf75 a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] 10.155.21.156,10.155.21.162 - - [10/Jun/2015 14:00:55] "GET /v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7 HTTP/1.1" 200 1073 0.080400
2015-06-10 14:00:55.356 8559 DEBUG routes.middleware [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Matched POST /df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7/action __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100
2015-06-10 14:00:55.357 8559 DEBUG routes.middleware [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Route path: '/{project_id}/volumes/:(id)/action', defaults: {'action': u'action', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7faef9da60d0>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102
2015-06-10 14:00:55.357 8559 DEBUG routes.middleware [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Match dict: {'action': u'action', 'controller': <cinder.api.openstack.wsgi.Resource object at 0x7faef9da60d0>, 'project_id': u'df97441007334e15a7e45e6564e6fee0', 'id': u'7f0ca54c-b481-4ab6-8534-969ceb3062e7'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103
2015-06-10 14:00:55.358 8559 INFO cinder.api.openstack.wsgi [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] POST http://10.155.21.150:8776/v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7/action
2015-06-10 14:00:55.359 8559 DEBUG cinder.api.openstack.wsgi [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] Action body: {"os-begin_detaching": null} get_method /usr/lib/python2.7/dist-packages/cinder/api/openstack/wsgi.py:1008
2015-06-10 14:00:55.395 8559 INFO cinder.api.openstack.wsgi [req-6efa46e9-8b61-44ea-bf45-a98ea004252f
a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] http://10.155.21.150:8776/v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7/action returned with HTTP 202
2015-06-10 14:00:55.396 8559 INFO eventlet.wsgi.server [req-6efa46e9-8b61-44ea-bf45-a98ea004252f a8e3e0c5875445faae2c10944600e77a df97441007334e15a7e45e6564e6fee0 - - -] 10.155.21.157,10.155.21.162 - - [10/Jun/2015 14:00:55] "POST /v1/df97441007334e15a7e45e6564e6fee0/volumes/7f0ca54c-b481-4ab6-8534-969ceb3062e7/action HTTP/1.1" 202 206 0.043197
Patch:
https:/