VMWare: vm spawn failure due to no attribute 'propSet' in concurrent case

[zhu zhu]

When doing a concurrent spawn for VMs(30). Noticed some VM spawn failures due to error below.

The reason is that during prebuild_instance, it will list all instances of existing. But if concurrent process for spawning is happening, it is possible that certain VM is just in process of creation in VCenter, and the vsphere SDK only return partial object for that VM. So there is no propSet for that VM created at the phase.

Should add check for these cases.

2014-09-14 03:53:31.311 2680 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher payload)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 282, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher pass
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 335, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 311, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2140, in run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher do_run_instance()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2139, in do_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher legacy_bdm_in_spec)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in _run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher notify("error", fault=e) # notify that build failed
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1206, in _run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher self._prebuild_instance(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in _prebuild_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher self._check_instance_exists(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in _check_instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher if self.driver.instance_exists(instance['name']):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 204, in instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return instance_id in self.list_instances()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 604, in list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher instances.extend(vmops.list_instances())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 2067, in list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher lst_vm_names = self._get_valid_vms_from_retrieve_result(vms)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1851, in _get_valid_vms_from_retrieve_result
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher for prop in vm.propSet:
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher AttributeError: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher

[Davanum Srinivas (DIMS) (dims-v)]

Zhu Zhu, will you be filing a review with a fix?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/123309

[Sean Dague (sdague)]

Patch stalled in merge conflict

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zhu Zhu (<email address hidden>) on branch: master
Review: https://review.openstack.org/123309

[Visnusaran Murugan (visnusaran-murugan)]

Raising an AttributeError is the right behavior IMO. The same issue happened under scale test, but for a VM deletion. A retry successfully deleted the error-ed VM.

The abandoned review returns a None in place of a missing propSet which is not doing to help complete VM operations like spawn or delete successfully.

[Visnusaran Murugan (visnusaran-murugan)]

_get_object_from_results initiates a ContinueRetrievePropertiesEx if the queried object is None. _get_object_for_value should check Token attribute in the resultSet and return None instead of accessing propSet which ends in an AttributeError.

Please ignore my earlier comment (#5)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/221038

[OpenStack Infra (hudson-openstack)]

Change abandoned by Visnusaran Murugan (<email address hidden>) on branch: master
Review: https://review.openstack.org/221038

[Visnusaran Murugan (visnusaran-murugan)]

This defect is handled in kilo+ code and affects Juno only. Refer to @garyk's comment in the review.

[Markus Zoeller (markus_z) (mzoeller)]

This is an automated cleanup. This bug report has been closed because it
is older than 18 months and there is no open code change to fix this.
After this time it is unlikely that the circumstances which lead to
the observed issue can be reproduced.

If you can reproduce the bug, please:
* reopen the bug report (set to status "New")
* AND add the detailed steps to reproduce the issue (if applicable)
* AND leave a comment "CONFIRMED FOR: <RELEASE_NAME>"
  Only still supported release names are valid (LIBERTY, MITAKA, OCATA, NEWTON).
  Valid example: CONFIRMED FOR: LIBERTY