VMware: NotAuthenticated occurred in the call to RetrievePropertiesEx

[Ryan Hsu]

The VMware Minesweeper CI occasionally runs into this error when trying to boot an instance:

2013-12-11 04:50:15.048 20785 DEBUG nova.virt.vmwareapi.driver [-] Task [ReconfigVM_Task] (returnval){
   value = "task-322"
   _type = "Task"
 } status: success _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:926
Reconfigured VM instance to enable vnc on port - 5986 _set_vnc_config /opt/stack/nova/nova/virt/vmwareapi/vmops.py:1461
Instance failed to spawn
Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1461, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 628, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 435, in spawn
    upload_folder, upload_name + ".vmdk")):
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1556, in _check_if_folder_file_exists
    "browser")
  File "/opt/stack/nova/nova/virt/vmwareapi/vim_util.py", line 173, in get_dynamic_property
    property_dict = get_dynamic_properties(vim, mobj, type, [property_name])
  File "/opt/stack/nova/nova/virt/vmwareapi/vim_util.py", line 179, in get_dynamic_properties
    obj_content = get_object_properties(vim, None, mobj, type, property_names)
  File "/opt/stack/nova/nova/virt/vmwareapi/vim_util.py", line 168, in get_object_properties
    options=options)
  File "/opt/stack/nova/nova/virt/vmwareapi/vim.py", line 187, in vim_request_handler
    fault_checker(response)
  File "/opt/stack/nova/nova/virt/vmwareapi/error_util.py", line 99, in retrievepropertiesex_fault_checker
    exc_msg_list))
VimFaultException: Error(s) NotAuthenticated occurred in the call to RetrievePropertiesEx

Full logs here for a CI build where this occurred are available here: http://162.209.83.206/logs/35303/31/

[Shawn Hartsock (hartsock)]

Marked High because this impacts the CI environment we use to approve other patches. If not for that context, I would mark this Medium.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/61555
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c47b4f388cb40d5792a206ac42ebdaebd5bde0d1
Submitter: Jenkins
Branch: master

commit c47b4f388cb40d5792a206ac42ebdaebd5bde0d1
Author: Sabari Kumar Murugesan <email address hidden>
Date: Wed Dec 11 12:11:40 2013 -0800

    VMware: use session.call_method to invoke api's

    All api invocations should use session.call_method, which has a
    retry mechanism to handle session failures. Few methods in vmops
    that are not using this wrapper to invoke api's, fail
    intermittently when the session is not authenticated.

    This patch fixes those methods. This issue is more pronounced
    during VMware CI (Minesweeper), where random tempest tests fail.

    The existing tests for vmops and those added for the retrying
    session in https://review.openstack.org/#/c/58890/ should cover
    this change, as we are re-routing the api invocation.

    Closes-Bug: #1260075
    Change-Id: I2beb7c92a4c3f53fd2120629e8f72bfd54d13740