OpenStack Compute (nova)

  1. Bug #1349888
  2. Comment #0

Comment 0 for bug 1349888

Revision history for this message
git-harry (git-harry) wrote : Attempting to attach the same volume multiple times can cause bdm record for existing attachment to be deleted.

nova assumes there is only ever one bdm per volume. When an attach is initiated a new bdm is created, if the attach fails a bdm for the volume is deleted however it is not necessarily the one that was just created. The following steps show how a volume can get stuck detaching because of this.

$ nova list
c+--------------------------------------+--------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
 +--------------------------------------+--------+--------+------------+-------------+------------------+
| cb5188f8-3fe1-4461-8a9d-3902f7cc8296 | test13 | ACTIVE | - | Running | private=10.0.0.2 |
 +--------------------------------------+--------+--------+------------+-------------+------------------+

$ cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | available | test10 | 1 | lvm1 | false | |
 +--------------------------------------+-----------+--------+------+-------------+----------+-------------+

$ nova volume-attach test13 c1e38e93-d566-4c99-bfc3-42e77a428cc4
+----------+--------------------------------------+
| Property | Value |
 +----------+--------------------------------------+
| device | /dev/vdb |
| id | c1e38e93-d566-4c99-bfc3-42e77a428cc4 |
| serverId | cb5188f8-3fe1-4461-8a9d-3902f7cc8296 |
| volumeId | c1e38e93-d566-4c99-bfc3-42e77a428cc4 |
 +----------+--------------------------------------+

$ cinder list
+--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | in-use | test10 | 1 | lvm1 | false | cb5188f8-3fe1-4461-8a9d-3902f7cc8296 |
 +--------------------------------------+--------+--------+------+-------------+----------+--------------------------------------+

$ nova volume-attach test13 c1e38e93-d566-4c99-bfc3-42e77a428cc4
ERROR (BadRequest): Invalid volume: status must be 'available' (HTTP 400) (Request-ID: req-1fa34b54-25b5-4296-9134-b63321b0015d)

$ nova volume-detach test13 c1e38e93-d566-4c99-bfc3-42e77a428cc4

$ cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
 +--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+
| c1e38e93-d566-4c99-bfc3-42e77a428cc4 | detaching | test10 | 1 | lvm1 | false | cb5188f8-3fe1-4461-8a9d-3902f7cc8296 |
 +--------------------------------------+-----------+--------+------+-------------+----------+--------------------------------------+

2014-07-29 14:47:13.952 ERROR oslo.messaging.rpc.dispatcher [req-134dfd17-14da-4de0-93fc-5d8d7bbf65a5 admin admin] Exception during message handling: <type 'NoneType'> can't be decoded
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 406, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 88, in wrapped
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher payload)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/exception.py", line 71, in wrapped
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 291, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher pass
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 277, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 319, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/excutils.py", line 82, in __exit__
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 307, in decorated_function
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4363, in detach_volume
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher self._detach_volume(context, instance, bdm)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/compute/manager.py", line 4309, in _detach_volume
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher connection_info = jsonutils.loads(bdm.connection_info)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/jsonutils.py", line 176, in loads
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher return json.loads(strutils.safe_decode(s, encoding), **kwargs)
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/nova/nova/openstack/common/strutils.py", line 134, in safe_decode
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher raise TypeError("%s can't be decoded" % type(text))
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher TypeError: <type 'NoneType'> can't be decoded
2014-07-29 14:47:13.952 31588 TRACE oslo.messaging.rpc.dispatcher