VMware: Cannot complete operation due to concurrent modification by another operation

[Ryan Hsu]

The VMware Minesweeper CI is occasionally seeing a "concurrent modification" failure when creating an instance. It's possible that this is the cause of a race condition in the VC Driver.

Task [CreateVM_Task] (returnval){
   value = "task-932"
   _type = "Task"
 } status: success _poll_task /opt/stack/nova/nova/virt/vmwareapi/driver.py:968

Created VM on the ESX host _execute_create_vm /opt/stack/nova/nova/virt/vmwareapi/vmops.py:305

Task [ReconfigVM_Task] (returnval){
   value = "task-939"
   _type = "Task"
 } status: error Cannot complete operation due to concurrent modification by another operation.

Fault Text not matched.

Setting instance vm_state to ERROR

Traceback (most recent call last):
  File "/opt/stack/nova/nova/compute/manager.py", line 5581, in _error_out_instance_on_exception
    yield
  File "/opt/stack/nova/nova/compute/manager.py", line 2562, in rebuild_instance
    self._rebuild_default_impl(**kwargs)
  File "/opt/stack/nova/nova/compute/manager.py", line 2417, in _rebuild_default_impl
    block_device_info=new_block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 619, in spawn
    admin_password, network_info, block_device_info)
  File "/opt/stack/nova/nova/virt/vmwareapi/vmops.py", line 593, in spawn
    root_gb_in_kb, linked_clone)
  File "/opt/stack/nova/nova/virt/vmwareapi/volumeops.py", line 75, in attach_disk_to_vm
    self._session._wait_for_task(reconfig_task)
  File "/opt/stack/nova/nova/virt/vmwareapi/driver.py", line 948, in _wait_for_task
    ret_val = done.wait()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 116, in wait
    return hubs.get_hub().switch()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 187, in switch
    return self.greenlet.switch()
VMwareDriverException: Cannot complete operation due to concurrent modification by another operation.

Full log for an affected run here: http://10.148.255.241/logs/83375/2/

[Ryan Hsu (rhsu)]

Also seen in: http://10.148.255.241/logs/88283/2/

[Ryan Hsu (rhsu)]

Also seen in http://10.148.255.241/logs/77408/10