With Ceph as cinder backend, detaching a volume doesn't work after the following steps:
# boot volume, create instance
nova boot --poll --key-name testkey --flavor 2 --image c848067a-a8bb-4631-a6a6-ed0404d444a9 test01
cinder create 1
export INSTANCEID=adcfcda6-5a27-4c85-a770-210a6b08894b
export VOLID=fad11ab9-3a1f-43d7-83c3-55bded6280e4
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state available $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
sleep 1
nova volume-attach $INSTANCEID $VOLID
sleep 1
cinder reset-state --state in-use $VOLID
sleep 1
nova volume-detach $INSTANCEID $VOLID
After the last step, the volume hangs in "detaching" state. nova-compute.log shows:
2015-03-17 08:06:06.456 1342 AUDIT nova.compute.manager [req-68662304-4140-4e19-b4dc-ec65cb43ec2b 04030aff6a0a4416bc532863109343ff 362d7cfe3e7b491290f7d1684db0a48f] [instance: adcfcda6-5a27-4c85-a770-210a6b08894b] Detach volume fad11ab9-3a1f-43d7-83c3-55bded6280e4 from mountpoint /dev/vdb
2015-03-17 08:06:06.574 1342 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher payload)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher pass
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher self._detach_volume(context, instance, bdm)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher connection_info = jsonutils.loads(bdm.connection_info)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return json.loads(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher return _default_decoder.decode(s)
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher TypeError: expected string or buffer
2015-03-17 08:06:06.574 1342 TRACE oslo.messaging.rpc.dispatcher
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] Returning exception expected string or buffer to caller
2015-03-17 08:06:06.575 1342 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib64/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib64/python2.6/site-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function\n pass\n', ' File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib64/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4267, in detach_volume\n self._detach_volume(context, instance, bdm)\n', ' File "/usr/lib64/python2.6/site-packages/nova/compute/manager.py", line 4212, in _detach_volume\n connection_info = jsonutils.loads(bdm.connection_info)\n', ' File "/usr/lib64/python2.6/site-packages/nova/openstack/common/jsonutils.py", line 164, in loads\n return json.loads(s)\n', ' File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads\n return _default_decoder.decode(s)\n', ' File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode\n obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n', 'TypeError: expected string or buffer\n']
The problem seems that the entry in the block_device_mapping table is removed for the attached volume which leads to this error.
