VMware: Unable to delete image cache vmdk while launching instance

[Ryan Hsu]

When running the Tempest tests in parallel, the following test is failing due to an instance spawning with ERROR:

  setUpClass (tempest.api.compute.admin.test_fixed_ips.FixedIPsTestJson)

The VimFaultException seen in the nova scheduler log is:

  Cannot delete file [datastore1] vmware_base/4f1cd69c-45c0-42d3-8ff1-33ff389fe5cc-flat.vmdk

Full traceback (non-wrapped here: http://paste.openstack.org/show/60501/):

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 1053, in _build_instance
    set_access_ip=set_access_ip)
  File "/opt/stack/nova/nova/compute/manager.py", line 356, in decorated_function
    return function(self, context, *args, **kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 1456, in _spawn
    LOG.exception(_('Instance failed to spawn'), instance=instance)
  File "/opt/stack/nova/nova/openstack/common/excutils.py", line 68, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/nova/nova/compute/manager.py", line 1453, in _spawn
    block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 587, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 458, in spawn
    dc_info.ref)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 171, in _delete_datastore_file
    file_delete_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 856, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
NovaException: Cannot delete file [datastore1] vmware_base/4f1cd69c-45c0-42d3-8ff1-33ff389fe5cc-flat.vmdk

[Ryan Hsu (rhsu)]

Note: This bug is a blocker for the VMware Minesweeper CI. Without it we are getting extremely long build times!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/69622

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/70137
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=fb12fa03ad21c4c8924175ae77bc6bd30139f089
Submitter: Jenkins
Branch: master

commit fb12fa03ad21c4c8924175ae77bc6bd30139f089
Author: Gary Kotton <email address hidden>
Date: Thu Jan 30 07:10:14 2014 -0800

    VMware: raise more specific exceptions

    In certain cases the exception handling for backend errors
    would be handled too broadly. This patch elevates the backend exception
    to the application so that it can treat specific errors.

    In the application we can now handle the following specific exceptions:
     - FileAlreadyExists - for example when moving a directory to a directory
                           that already exists
     - InvalidProperty - for example when using neutron and opaque networks
                         are not supported
     - AlreadyExists - for example a port group already exists
     - NotAuthenticated - for example the operation is denied as because a
                          session is not established

    Related-Bug: #1266579
    Related-Bug: #1266580
    Closes-bug: #1276539

    Co-authored-by: Matthew Booth <email address hidden>
    Change-Id: I0873f570d97345208c8d9b5bcbb5a980a7a58f9d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/69622
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=716b426eccd6f84a9c4499031ace950916215ac7
Submitter: Jenkins
Branch: master

commit 716b426eccd6f84a9c4499031ace950916215ac7
Author: Gary Kotton <email address hidden>
Date: Tue Jan 28 07:03:38 2014 -0800

    VMware: prevent race for vmdk deletion

    Prevent a race condition for the deletion of a file when there
    is more than one process trying to erase the file.

    In the application we can now handle the following specific exceptions:
     - CannotDeleteFile - the file cannot be deleted
     - FileFault - a file access exception
     - FileLocked - an attempt is made to lock a file that is
                    already in use
     - FileNotFound - the specific file does not exist

    Change-Id: I58e07b9e630c063f894cbbb138612a39e912acd5
    Closes-bug: #1266580