Steps are as following:
1) Create one VM
2) Attach volume to the VM
3) pause the VM
4) detach the volume
5) unpause the VM
6) re-attch the VM to same device, nova compute throw exception
2013-10-20 23:21:22.520 DEBUG amqp [-] Channel open from (pid=19728) _open_ok /usr/local/lib/python2.7/dist-packages/amqp-1.0.12-py2.7.egg/amqp/channel.py:420
2013-10-20 23:21:22.520 ERROR nova.openstack.common.rpc.amqp [req-5f0d786e-1273-4611-b0a5-a787754c6bc8 admin admin] Exception during message handling
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 244, in decorated_function
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp pass
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 230, in decorated_function
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 272, in decorated_function
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 259, in decorated_function
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3649, in attach_volume
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3644, in attach_volume
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3690, in _attach_volume
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp connector)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3680, in _attach_volume
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp encryption=encryption)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 1107, in attach_volume
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp raise exception.DeviceIsBusy(device=disk_dev)
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp DeviceIsBusy: The supplied device (vdb) is busy.
2013-10-20 23:21:22.520 TRACE nova.openstack.common.rpc.amqp
^C2013-10-20 23:21:24.871 INFO nova.openstack.common.service [-] Caught SIGINT, exiting
I do not believe this bug can be easily fixed.
The reason you are getting the DeviceIsBusy exception is because Linux sysfs did not receive the hotplug event (uevent) when the volume/device was detached. Since the instance was paused, no uevents could be created and so, Linux still believes the device is still there. When cinder attempts to hotplug the volume, the device name that nova chooses is found using by a database lookup of which devices are attached to the instance and selecting the next available device name to use. The device nova picks is /dev/vdb, the same one Linux sysfs believes is still attached to it. Therefore, device hotplug fails.
A possible workaround is to reboot Linux in order to force a sysfs refresh and free up the device name.
A possible solution is to prevent users from trying to hotplug/unplug to a paused/suspended instance, since it would corrupt sysfs.
Any thoughts?