nova assumes there is only ever one bdm per volume. When an attach is initiated a new bdm is created, if the attach fails a bdm for the volume is deleted however it is not necessarily the one that was just created. The following steps show how a volume can get stuck detaching because of this.
$ nova list
c+--------------------------------------+--------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+--------+--------+------------+-------------+------------------+
| cb5188f8-3fe1-4461-8a9d-3902f7cc8296 | test13 | ACTIVE | - | Running | private=10.0.0.2 |
+--------------------------------------+--------+--------+------------+-------------+------------------+
$ cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | available | test10 | 1 | lvm1 | false | |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
$ cinder list
+--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | in-use | test10 | 1 | lvm1 | false | cb5188f8-3fe1-4461-8a9d-3902f7cc8296 |
+--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+
$ nova volume-attach test13 c1e38e93-d566-4c99-bfc3-42e77a428cc4
ERROR (BadRequest): Invalid volume: status must be 'available' (HTTP 400) (Request-ID: req-1fa34b54-25b5-4296-9134-b63321b0015d)
$ nova volume-detach test13 c1e38e93-d566-4c99-bfc3-42e77a428cc4
$ cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | detaching | test10 | 1 | lvm1 | false | cb5188f8-3fe1-4461-8a9d-3902f7cc8296 |
+--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+
2014-07-29 14:47:13.952 ERROR oslo.messaging.rpc.dispatcher [req-134dfd17-14da-4de0-93fc-5d8d7bbf65a5 admin admin] Exception during message handling: <type 'NoneType'> can't be decoded
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 406, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 291, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 277, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 319, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 307, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4363, in detach_volume
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher self._detach_volume(context, instance, bdm)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4309, in _detach_volume
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher connection_info = jsonutils.loads(bdm.connection_info)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 176, in loads
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return json.loads(strutils.safe_decode(s, encoding), **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/strutils.py", line 134, in safe_decode
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher raise TypeError("%s can't be decoded" % type(text))
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher TypeError: <type 'NoneType'> can't be decoded
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher
nova assumes there is only ever one bdm per volume. When an attach is initiated a new bdm is created, if the attach fails a bdm for the volume is deleted however it is not necessarily the one that was just created. The following steps show how a volume can get stuck detaching because of this.
$ nova list ------- ------- ------- ------- -----+- ------- +------ --+---- ------- -+----- ------- -+----- ------- ------+ ------- ------- ------- ------- ----+-- ------+ ------- -+----- ------- +------ ------- +------ ------- -----+ 3fe1-4461- 8a9d-3902f7cc82 96 | test13 | ACTIVE | - | Running | private=10.0.0.2 | ------- ------- ------- ------- ----+-- ------+ ------- -+----- ------- +------ ------- +------ ------- -----+
c+-----
| ID | Name | Status | Task State | Power State | Networks |
+------
| cb5188f8-
+------
$ cinder list ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- -+ ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- -+ d566-4c99- bfc3-42e77a428c c4 | available | test10 | 1 | lvm1 | false | | ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- -+
+------
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| c1e38e93-
+------
$ nova volume-attach test13 c1e38e93- d566-4c99- bfc3-42e77a428c c4 ----+-- ------- ------- ------- ------- ------- -+ ----+-- ------- ------- ------- ------- ------- -+ d566-4c99- bfc3-42e77a428c c4 | 3fe1-4461- 8a9d-3902f7cc82 96 | d566-4c99- bfc3-42e77a428c c4 | ----+-- ------- ------- ------- ------- ------- -+
+------
| Property | Value |
+------
| device | /dev/vdb |
| id | c1e38e93-
| serverId | cb5188f8-
| volumeId | c1e38e93-
+------
$ cinder list ------- ------- ------- ------- ----+-- ------+ ------- -+----- -+----- ------- -+----- -----+- ------- ------- ------- ------- ------- --+ ------- ------- ------- ------- ----+-- ------+ ------- -+----- -+----- ------- -+----- -----+- ------- ------- ------- ------- ------- --+ d566-4c99- bfc3-42e77a428c c4 | in-use | test10 | 1 | lvm1 | false | cb5188f8- 3fe1-4461- 8a9d-3902f7cc82 96 | ------- ------- ------- ------- ----+-- ------+ ------- -+----- -+----- ------- -+----- -----+- ------- ------- ------- ------- ------- --+
+------
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| c1e38e93-
+------
$ nova volume-attach test13 c1e38e93- d566-4c99- bfc3-42e77a428c c4 25b5-4296- 9134-b63321b001 5d)
ERROR (BadRequest): Invalid volume: status must be 'available' (HTTP 400) (Request-ID: req-1fa34b54-
$ nova volume-detach test13 c1e38e93- d566-4c99- bfc3-42e77a428c c4
$ cinder list ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- ------- ------- ------- -----+ ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- ------- ------- ------- -----+ d566-4c99- bfc3-42e77a428c c4 | detaching | test10 | 1 | lvm1 | false | cb5188f8- 3fe1-4461- 8a9d-3902f7cc82 96 | ------- ------- ------- ------- ----+-- ------- --+---- ----+-- ----+-- ------- ----+-- ------- -+----- ------- ------- ------- ------- -----+
+------
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+------
| c1e38e93-
+------
2014-07-29 14:47:13.952 ERROR oslo.messaging. rpc.dispatcher [req-134dfd17- 14da-4de0- 93fc-5d8d7bbf65 a5 admin admin] Exception during message handling: <type 'NoneType'> can't be decoded rpc.dispatcher Traceback (most recent call last): rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 134, in _dispatch_and_reply rpc.dispatcher incoming.message)) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 177, in _dispatch rpc.dispatcher return self._do_ dispatch( endpoint, method, ctxt, args) rpc.dispatcher File "/usr/local/ lib/python2. 7/dist- packages/ oslo/messaging/ rpc/dispatcher. py", line 123, in _do_dispatch rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 406, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ exception. py", line 88, in wrapped rpc.dispatcher payload) rpc.dispatcher File "/opt/stack/ nova/nova/ openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ exception. py", line 71, in wrapped rpc.dispatcher return f(self, context, *args, **kw) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 291, in decorated_function rpc.dispatcher pass rpc.dispatcher File "/opt/stack/ nova/nova/ openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 277, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 319, in decorated_function rpc.dispatcher kwargs['instance'], e, sys.exc_info()) rpc.dispatcher File "/opt/stack/ nova/nova/ openstack/ common/ excutils. py", line 82, in __exit__ rpc.dispatcher six.reraise( self.type_ , self.value, self.tb) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 307, in decorated_function rpc.dispatcher return function(self, context, *args, **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 4363, in detach_volume rpc.dispatcher self._detach_ volume( context, instance, bdm) rpc.dispatcher File "/opt/stack/ nova/nova/ compute/ manager. py", line 4309, in _detach_volume rpc.dispatcher connection_info = jsonutils. loads(bdm. connection_ info) rpc.dispatcher File "/opt/stack/ nova/nova/ openstack/ common/ jsonutils. py", line 176, in loads rpc.dispatcher return json.loads( strutils. safe_decode( s, encoding), **kwargs) rpc.dispatcher File "/opt/stack/ nova/nova/ openstack/ common/ strutils. py", line 134, in safe_decode rpc.dispatcher raise TypeError("%s can't be decoded" % type(text)) rpc.dispatcher TypeError: <type 'NoneType'> can't be decoded rpc.dispatcher
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.