VMware: Unable to launch instance because vmware_base already exists

[Ryan Hsu]

When running the Tempest tests in parallel, 2 tests are failing due to instances spawning with ERROR:

  setUpClass (tempest.api.compute.admin.test_fixed_ips.FixedIPsTestXml)
  setUpClass (tempest.api.compute.admin.test_fixed_ips_negative.FixedIPsNegativeTestJson)

The VimFaultException seen in the nova scheduler log is:

  Cannot complete the operation because the file or folder [datastore1] vmware_base already exists

Full Traceback here (non-wrapped version here: http://paste.openstack.org/show/60502/):

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1053, in _build_instance
    set_access_ip=set_access_ip)
  File "/opt/stack/nova/nova/compute/manager.py", line 356, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 1456, in _spawn
    LOG.exception(_('Instance failed to spawn'), instance=instance)
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 1453, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 587, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 437, in spawn
    upload_folder, upload_name + ".vmdk")):
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1549, in _check_if_folder_file_exists
    ds_ref)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 1534, in _mkdir
    createParentDirectories=False)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 795, in _call_method
    return temp_module(*args, **kwargs)
  File "/opt/stack/nova/nova/virt/vmwareapi/vim.py", line 200, in vim_request_handler
    raise error_util.VimFaultException(fault_list, excep)
VimFaultException: Server raised fault: 'Cannot complete the operation because the file or folder [datastore1] vmware_base already exists'

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/65306

[Ryan Hsu (rhsu)]

Note: This bug is a blocker for the VMware Minesweeper CI. Without it we are getting extremely long build times!

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70137
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fb12fa03ad21c4c8924175ae77bc6bd30139f089
Submitter: Jenkins
Branch: master

commit fb12fa03ad21c4c8924175ae77bc6bd30139f089
Author: Gary Kotton <email address hidden>
Date: Thu Jan 30 07:10:14 2014 -0800

    VMware: raise more specific exceptions

    In certain cases the exception handling for backend errors
    would be handled too broadly. This patch elevates the backend exception
    to the application so that it can treat specific errors.

    In the application we can now handle the following specific exceptions:
     - FileAlreadyExists - for example when moving a directory to a directory
                           that already exists
     - InvalidProperty - for example when using neutron and opaque networks
                         are not supported
     - AlreadyExists - for example a port group already exists
     - NotAuthenticated - for example the operation is denied as because a
                          session is not established

    Related-Bug: #1266579
    Related-Bug: #1266580
    Closes-bug: #1276539

    Co-authored-by: Matthew Booth <email address hidden>
    Change-Id: I0873f570d97345208c8d9b5bcbb5a980a7a58f9d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/65306
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=743b74d59d3279478c512a0a559642b1d05284bc
Submitter: Jenkins
Branch: master

commit 743b74d59d3279478c512a0a559642b1d05284bc
Author: Gary Kotton <email address hidden>
Date: Tue Jan 7 08:32:44 2014 -0800

    VMware: fix race for datastore directory existence

    Treat the exception that is raised when creating a directory
    and it already exists. This may happen due to two processes trying
    to create the same directory at the same time.

    Change-Id: Ia0ebd674345734e7cfa31ccd400fdba93646c554
    Closes-bug: #1266579