VMware: attaching a volume to the VM failed

[Vincent Hou]

I was using the image trend-tinyvm1-flat.vmdk. When I boot an VM and attach a volume to it. The following error message was shown:

================================n-cpu==================================

2013-09-17 06:47:05.669 ERROR nova.openstack.common.rpc.amqp [req-fc4e0a9e-6bd5-45ac-b841-c89e17394603 admin demo] Exception during me
ssage handling
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in
_process_data
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp **args)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 17
2, in dispatch
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 89, in wrapped
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp payload)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_
function
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp pass
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_function
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_function
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_function
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3577, in attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3572, in attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3614, in _attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp connector)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3604, in _attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp mountpoint)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 290, in attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp mountpoint)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 272, in attach_volume
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp self._attach_volume_vmdk(connection_info, instance, mountpoint)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 222, in _attach_volume_vmdk
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp unit_number=unit_number)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 71, in attach_disk_to_vm
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp self._session._wait_for_task(instance_uuid, reconfig_task)
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 779, in _wait_for_task
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp ret_val = done.wait()
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp return hubs.get_hub().switch()
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp return self.greenlet.switch()
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp NovaException: The attempted operation cannot be performed in the current state (Powered on).
2013-09-17 06:47:05.669 TRACE nova.openstack.common.rpc.amqp

[Gary Kotton (garyk)]

Hi,
Can you please try this again. The vmware cinder support was approved on Sept 5th.
In addition to this can you please indicate which cinder backend you used.
Thanks
Gary

[Vincent Hou (houshengbo)]

Hi Gary,
I tried today. There is still errors when I attach the volume to the VM.

============================n-cpu=========================

2013-10-11 06:15:48.317 ERROR nova.openstack.common.rpc.amqp [req-1652a9a6-a236-49b0-8e83-063612b1f5a0 admin demo] Exception during me
ssage handling
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in
_process_data
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 17
2, in dispatch
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp pass
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3640, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3635, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3681, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp connector)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3671, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp encryption=enc...

[Vincent Hou (houshengbo)]

Hi Gary,
I tried it today. There is still errors when I attach the volume to the VM. The image I use is trend-tinyvm1-flat.vmdk.
============================n-cpu=========================
2013-10-11 06:15:48.317 ERROR nova.openstack.common.rpc.amqp [req-1652a9a6-a236-49b0-8e83-063612b1f5a0 admin demo] Exception during me
ssage handling
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 461, in
_process_data
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp **args)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 17
2, in dispatch
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 90, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp payload)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/exception.py", line 73, in wrapped
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return f(self, context, *args, **kw)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 243, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp pass
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 229, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 271, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp e, sys.exc_info())
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 258, in decorated_
function
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp return function(self, context, *args, **kwargs)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3640, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp context, instance, mountpoint)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3635, in attach_vo
lume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp mountpoint, instance)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3681, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp connector)
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/compute/manager.py", line 3671, in _attach_v
olume
2013-10-11 06:15:48.317 TRACE nova.openstack.common.rpc.amqp encryption=encryption)
2013-10-11 06:15:48....

[Kartik Bommepally (kartikaditya)]

Hi Vincent,

Can you give some more details on you setup:
1. Have you configured with ESX/VC?
2. Which version of VMware server are you using.

Can you try powering off the instance and then perform a volume attach.

P.S: You could try searching for "The attempted operation cannot be performed in the current state (Powered on)" in http://kb.vmware.com/ if it helps.

[Gary Kotton (garyk)]

The problem seems to arise from the fact that a sparse disk type is used. We are investigating

[Kartik Bommepally (kartikaditya)]

The issue is when the instance uses an IDE controller.

IDE controller does not support hot adding of a virtual disk http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025883

However if an IDE controller is not being used for the primary disk of the virtual machine then the issue must not arise.

One possible fix would be to use a SCSI controller (one of http://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.wssdk.apiref.doc%2Fvim.vm.device.VirtualSCSIController.html) for volume's disk irrespective of primary disk's controller.

[ChangBo Guo(gcb) (glongwave)]

These are configuration options:
vCenter: 5.5
vSphere web service SDK 5.5
Image:
# wget http://partnerweb.vmware.com/programs/vmdkimage/trend-tinyvm1-flat.vmdk
# glance image-create --name trend-thin --is-public=True --container-format=bare --disk-format=vmdk --property vmware_disktype="thin" --property vmware_adaptertype="ide" < trend-tinyvm1-flat.vmdk

we get the error "2013-11-18 21:31:11.859 18737 TRACE nova.openstack.common.rpc.amqp NovaException: The attempted operation cannot be performed in the current state (Powered on)."

The root cause is that some disks are not hot-addable, same we met in bug [1]

The possible fix is stop instance before attach/detach volume. Note this will affect user's normal workload. like we did for bug [1], or mark this won't fix and suggest user to use SCSI controller in comment #6

 [1] https://bugs.launchpad.net/nova/+bug/1247556

[ChangBo Guo(gcb) (glongwave)]

This depends instance's image. Maybe we need raise a exception to tell user what 's happening.

[kaitian521 (kaitian521)]

   glance add name="ubuntuLTS" disk_format=vmdk container_format=ovf \
is_public=true vmware_adaptertype="lsiLogic" vmware_disktype="preallocated" \
vmware_ostype="ubuntu64Guest" < ubuntuLTS-flat.vmdk

you can use this command to import an image,

you can attach a volume in ACTIVE

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65799

[Tracy Jones (tjones-i)]

all reviews are abandoned for this and it has not been updated in a long time. moving back to triaged. Please set to in progress if you plan on working on it

[Vipin Balachandran (vbala)]

This is not a valid cinder bug. Nova vmware driver has to create or use an existing controller in the instance VM to attach the volume and it is unrelated to the controller in volume backing. Please re-open this if you disagree.

[Tracy Jones (tjones-i)]

raising prio as it is blocking some of the cinder test cases.

[Thang Pham (thang-pham)]

The issue is pointed out above, i.e. ide controller does not support hotplugging. To fix this, an exception should be thrown when attempting to attach a volume to an active instance (as pointed out in the review by danpb).

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/122251

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/122251
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=e3fd9b377c082791670ce94ed972e0103f88d399
Submitter: Jenkins
Branch: master

commit e3fd9b377c082791670ce94ed972e0103f88d399
Author: Thang Pham <email address hidden>
Date: Wed Sep 17 17:14:15 2014 -0400

    VMware: Support volume hotplug

    When a volume is attached to an instance, the disk_type and
    adapter_type are found by making a get_dynamic_property vim
    call to the instance. If the adapter_type returned is
    "ide", the volume attach fails with "operation cannot be
    performed in the current state". This is because VMware IDE
    adapter does not support disk hotplug.

    Instead of making a get_dynamic_property vim call to the
    instance, it should be made to the volume (shadow VM). This
    properly indicates what adapter_type is needed by the volume.
    If the instance does not have a controller to support the
    volume adapter_type (e.g. lsiLogic), one will be added.
    If the volume adapter_type is "ide", a exception.Invalid will
    be thrown.

    Change-Id: Id826a33298885e9dbd4b52c9fe518ed2e6d50469
    Closes-Bug: #1226543