VMware driver: Creating a volume does not try to find datastores on all hosts available in VC

[Subbu]

Using cinder.volume.drivers.vmware.vmdk.VMwareVcVmdkDriver against a VC with three hosts (host-10, host-12 and host-19). Creating a volume tries to look for datastores only in the first host and ignores the rest as shown in the log below.

2013-09-24 15:34:16.369 15048 INFO cinder.volume.drivers.vmware.vmdk [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Backing not found, creating for volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512
2013-09-24 15:34:16.456 15048 WARNING cinder.volume.drivers.vmware.vmdk [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Unable to find suitable datastore for volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512 under host: (obj){
   value = "host-10"
   _type = "HostSystem"
 }. More details: No datastores
2013-09-24 15:34:16.478 15048 WARNING cinder.volume.drivers.vmware.vmdk [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Unable to find suitable datastore for volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512 under host: (obj){
   value = "host-10"
   _type = "HostSystem"
 }. More details: No datastores
2013-09-24 15:34:16.504 15048 WARNING cinder.volume.drivers.vmware.vmdk [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Unable to find suitable datastore for volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512 under host: (obj){
   value = "host-10"
   _type = "HostSystem"
 }. More details: No datastores
2013-09-24 15:34:16.559 15048 ERROR cinder.volume.drivers.vmware.vmdk [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Unable to create volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512 on the hosts: [(ObjectContent){
   obj =
      (obj){
         value = "host-10"
         _type = "HostSystem"
      }
   propSet[] =
      (DynamicProperty){
         name = "name"
         val = "10.112.184.131"
      },
 }, (ObjectContent){
   obj =
      (obj){
         value = "host-12"
         _type = "HostSystem"
      }
   propSet[] =
      (DynamicProperty){
         name = "name"
         val = "10.112.184.16"
      },
 }, (ObjectContent){
   obj =
      (obj){
         value = "host-19"
         _type = "HostSystem"
      }
   propSet[] =
      (DynamicProperty){
         name = "name"
         val = "10.112.187.62"
      },
 }].
2013-09-24 15:34:16.631 15048 ERROR cinder.openstack.common.rpc.amqp [req-ffdb25ba-1e88-4f23-9df7-d09e525e4c33 a038902df2744c01b7485bc8595925eb e0ef1acf1b9a425f9e84da3a82a5d80e] Exception during message handling
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp Traceback (most recent call last):
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/openstack/common/rpc/amqp.py", line 441, in _process_data
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp **args)
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/openstack/common/rpc/dispatcher.py", line 148, in dispatch
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/volume/manager.py", line 512, in copy_volume_to_image
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp payload['message'] = unicode(error)
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp self.gen.next()
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/volume/manager.py", line 506, in copy_volume_to_image
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp image_meta)
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/volume/drivers/vmware/vmdk.py", line 683, in copy_volume_to_image
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp backing = self._create_backing_in_inventory(volume)
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp File "/home/subbu/review/cinder/cinder/volume/drivers/vmware/vmdk.py", line 342, in _create_backing_in_inventory
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp {'vol': volume['name'], 'hosts': hosts})
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp VimException: Unable to create volume: volume-17760ecc-2796-42e2-875a-5e65a23ec512 on the hosts: [(ObjectContent){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp obj =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (obj){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp value = "host-10"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp _type = "HostSystem"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp propSet[] =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (DynamicProperty){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp name = "name"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp val = "10.112.184.131"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp },
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }, (ObjectContent){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp obj =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (obj){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp value = "host-12"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp _type = "HostSystem"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp propSet[] =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (DynamicProperty){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp name = "name"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp val = "10.112.184.16"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp },
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }, (ObjectContent){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp obj =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (obj){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp value = "host-19"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp _type = "HostSystem"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp propSet[] =
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp (DynamicProperty){
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp name = "name"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp val = "10.112.187.62"
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp },
2013-09-24 15:34:16.631 15048 TRACE cinder.openstack.common.rpc.amqp }].

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/48263

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/47130
Committed: http://github.com/openstack/cinder/commit/010fe685e50a3680d8c8b9b6602677df90e815b3
Submitter: Jenkins
Branch: master

commit 010fe685e50a3680d8c8b9b6602677df90e815b3
Author: Kartik Bommepally <email address hidden>
Date: Mon Sep 16 03:23:23 2013 -0700

    VMware: Usng RetrvProprtisEx & does multi ESX scan

    The existing code uses deprecated API RetrieveProperties for querying through
    property collector. The public doc asks to use RetrievePropertiesEx instead.

    RetrievePropertiesEx works by retrieving results in serveral batches rather than
    retrieval in a single API call. This works well in environments containing
    large number of VMs or ESX hosts.

    There are three steps involved 1. Get first batch 2. Continue retrieving using
    a token 3. Cancel retrieval when not needed

    Batch size is user configurable, default is 100 results per batch.

    The review also fixes a careless programming error leading to scanning only
    1 ESX host under the vCenter server for datastores.

    Fixes bug: 1227078
    Fixes bug: 1229654
    DocImpact

    Change-Id: I85a6f6a7e2d764b9843efbd0dd347fba3ef19310