vmware: Error observed while creating volume from existing glance image-id on vsan datastore.

[satyadev svn]

Create profile -say gold which mapped to vsan data store.
Now create a volume from existing glance image-id - by using bleow commands

ssatya@devstack:~/devstack$ nova image-list
+--------------------------------------+------------------------+--------+--------+
| ID | Name | Status | Server |
+--------------------------------------+------------------------+--------+--------+
| ff0280b5-1174-42c6-9ce4-2a6e7334dd32 | cirros-0.3.0-i386-disk | ACTIVE | |
+--------------------------------------+------------------------+--------+--------+
ssatya@devstack:~/devstack$ cinder create --image-id ff0280b5-1174-42c6-9ce4-2a6e7334dd32 --name vol_def_vsan 1 --volume-type gold

observation : we were able to create volumes from above image id on nfs, vmfs data stores

Please see attached file "volume_from_image_vsan.log" for more details.

Log trace :
=======================================================================
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api
2014-04-03 15:36:37.054 ERROR cinder.volume.drivers.vmware.api [-] Task: (returnval){
   value = "task-106"
   _type = "Task"
 } failed with error: File [vsanDatastore] 4e363d53-07a1-43b1-e595-005056876afe/volume-6962ac2b-60d3-4595-8986-b27ea87744c0-flat.vmdk was not found.
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api Traceback (most recent call last):
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 276, in _poll_task
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api raise error_util.VimFaultException([], error_msg)
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api VimFaultException: File [vsanDatastore] 4e363d53-07a1-43b1-e595-005056876afe/volume-6962ac2b-60d3-4595-8986-b27ea87744c0-flat.vmdk was not found
2014-04-03 15:36:37.054 TRACE cinder.volume.drivers.vmware.api
2014-04-03 15:36:37.055 ERROR cinder.openstack.common.loopingcall [-] in fixed duration looping call
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall Traceback (most recent call last):
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall File "/opt/stack/cinder/cinder/openstack/common/loopingcall.py", line 76, in _inner
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall self.f(*self.args, **self.kw)
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall File "/opt/stack/cinder/cinder/volume/drivers/vmware/api.py", line 280, in _poll_task
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall raise excep
2014-04-03 15:36:37.055 TRACE cinder.openstack.common.loopingcall VimFaultException: File [vsanDatastore] 4e363d53-07a1-43b1-e595-005056876afe/volume-6962ac2b-60d3-4595-8986-b27ea87744c0-flat.vmdk was not found
=======================================================================

2014-04-03 15:37:14.150 ERROR oslo.messaging._drivers.common [req-1b9e96f5-ed37-400c-91c3-a54f373561cc a118de5f08f1426593b7ab32a18ddcc7 ac2a6c58c8a14f60beeed703cf5586f6] ['Traceback (most recent call last):\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 36, in _dispatch\n incoming.reply(self.callback(incoming.ctxt, incoming.message))\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in __call__\n return self._dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 104, in _dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 363, in create_volume\n _run_flow()\n', ' File "/opt/stack/cinder/cinder/volume/manager.py", line 356, in _run_flow\n flow_engine.run()\n', ' File "/opt/stack/taskflow/taskflow/utils/lock_utils.py", line 54, in wrapper\n return f(*args, **kwargs)\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 94, in run\n self._run()\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/engine.py", line 109, in _run\n misc.Failure.reraise_if_any(failures.values())\n', ' File "/opt/stack/taskflow/taskflow/utils/misc.py", line 586, in reraise_if_any\n failures[0].reraise()\n', ' File "/opt/stack/taskflow/taskflow/utils/misc.py", line 593, in reraise\n six.reraise(*self._exc_info)\n', ' File "/opt/stack/taskflow/taskflow/engines/action_engine/executor.py", line 34, in _execute_task\n result = task.execute(**arguments)\n', ' File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 594, in execute\n **volume_spec)\n', ' File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 556, in _create_from_image\n image_id, image_location, image_service)\n', ' File "/opt/stack/cinder/cinder/volume/flows/manager/create_volume.py", line 470, in _copy_image_to_volume\n raise exception.ImageCopyFailure(reason=ex)\n', 'ImageCopyFailure: Failed to copy image to volume: Bad or unexpected response from the storage volume backend API: Exception in copy_image_to_volume: File [vsanDatastore] 4e363d53-07a1-43b1-e595-005056876afe/volume-6962ac2b-60d3-4595-8986-b27ea87744c0-flat.vmdk was not found. Deleting the backing: (result){\n value = "vm-70"\n _type = "VirtualMachine"\n }.\n']
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenpool.py", line 80, in _spawn_n_impl
    func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_executors/base.py", line 46, in _dispatch
    incoming.reply(failure=exc_info)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 63, in reply
    self._send_reply(conn, reply, failure, log_failure=log_failure)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 44, in _send_reply
    log_failure)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/common.py", line 327, in serialize_remote_exception
    json_data = jsonutils.dumps(data)
  File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/openstack/common/jsonutils.py", line 164, in dumps
    return json.dumps(value, default=default, **kwargs)
  File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
    return _iterencode(o, 0)
ValueError: Circular reference detected
======================================================================

[Vipin Balachandran (vbala)]

Only images with vmware_disktype=streamOptimized can be copied to vSAN datastores. We shouldn't select vSAN datastores for volume creation if vmware_disktype=streamOptimized.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/113862

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/113862
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=2c672d1100ad4f44517838e17156b3ea6300b1cc
Submitter: Jenkins
Branch: master

commit 2c672d1100ad4f44517838e17156b3ea6300b1cc
Author: Vipin Balachandran <email address hidden>
Date: Fri Aug 22 19:23:11 2014 +0530

    VMware: Improve datastore selection logic

    The current datastore selection logic is not modularized and difficult
    to extend. The retype API implementation has a requirement to specify
    hard anti-affinity requirement with the current backing datastore. Some
    of the bug fixes also need to specify hard affinity with one or more
    datastore types. To support such requirements and to enable future
    extensions, this patch introduces a new module which contains datastore
    selection logic. The existing code for datastore selection is reused as
    much as possible. The dependency on existing datastore selection logic
    will be removed in a separate patch.

    The current datastore selection iterates over a list of hosts, for each
    host, queries the connected valid datastores and tries to select a
    suitable datastore. The filtering is based on space, storage profile,
    number of connected hosts and space utilization. The space utilization
    is used only for breaking ties. If a suitable datastore is found, further
    processing of list of hosts is skipped, which could result in uneven space
    utilization. To solve this, the new selection logic introduces a requirement
    called 'preferred_utilization_threshold' which can be exposed as a driver
    config option.

    Partial-bug: #1275682
    Partial-bug: #1301943
    Partial-bug: #1293955

    Change-Id: I17e90aa09a303fbb8d4ad90037f440c8c4e7d072

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156904

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/156904
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=753d6a8574fcc2c0680c9f75c31e7bc9035e1534
Submitter: Jenkins
Branch: master

commit 753d6a8574fcc2c0680c9f75c31e7bc9035e1534
Author: Vipin Balachandran <email address hidden>
Date: Wed Jan 7 11:39:43 2015 +0530

    VMware: Skip vSAN for preallocated image download

    Copying preallocated image to volume involves creating
    a virtual disk using the image. The virtual disk has a
    descriptor and an extent. The preallocated image is used
    as the flat extent to create the virtual disk.

    To create virtual disk from preallocated image, we first
    create a temporary virtual disk (with a descriptor and
    flat extent), then replace its flat extent with the
    downloaded image. Since vSAN datastores do not support
    flat extent, the above operation fails if such a datastore
    is selected for virtual disk creation. This patch fixes
    the problem by not selecting vSAN datastores for temporary
    virtual disk creation.

    Change-Id: Ib4c0fcf5c379b137e70378f137d996bf510f05b3
    Closes-Bug: #1301943