Nova boot server failed due to exception "Cannot complete the operation because the file or folder .vmdk already exists "

[kirandevraaj]

Nova boot server failed due to exception "vmware_base/373ca119-5b29-441d-bd9c-b2ccfb4444ad/373ca119-5b29-441d-bd9c-b2ccfb4444ad.20.vmdk already exists"

Environment -
Openstack Pike/Queens/Master
KVM computes - 4
VCenter + ESXi computes - 6

nova/conductor.log.1:2018-02-01 18:48:42.926 5826 ERROR nova.scheduler.utils [req-4bcfad48-2a15-4aa4-8c32-ebb3798a8b09 47b7fb99c82c413caae4081c35b7456a f2ab30de47df41e488a77258c7a1c6a1 - default default] [instance: 1c4830b5-f1f5-4bfb-bcd9-63c67aed127f] Error from last host: oscompute01 (node domain-c74754.bcea1667-7332-4f99-8aae-3f2bc39c2fc9): [u'Traceback (most recent call last):\n', u' File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 1856, in _do_build_and_run_instance\n filter_properties)\n', u' File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2086, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 1c4830b5-f1f5-4bfb-bcd9-63c67aed127f was re-scheduled: Cannot complete the operation because the file or folder [datastore-nfs] vmware_base/373ca119-5b29-441d-bd9c-b2ccfb4444ad/373ca119-5b29-441d-bd9c-b2ccfb4444ad.20.vmdk already exists\n']

neutron server log,
neutron/server.log.2:2018-02-01 18:48:53.788 5838 DEBUG neutron.notifiers.nova [-] Sending events: [{'tag': u'4c311f56-a73f-47a6-aa58-95f1a774396e', 'name': 'network-vif-deleted', 'server_uuid': u'1c4830b5-f1f5-4bfb-bcd9-63c67aed127f'}] send_events /opt/mhos/openstack/neutron/lib/python2.7/site-packages/neutron/notifiers/nova.py:242
neutron/server.log.2:2018-02-01 18:48:54.317 5838 DEBUG novaclient.v2.client [-] REQ: curl -g -i -X POST http://192.168.7.7:8774/v2.1/86d8652e2b834d29a744c6c66a5fb65e/os-server-external-events -H "User-Agent: python-novaclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-OpenStack-Nova-API-Version: 2.1" -H "X-Auth-Token: {SHA1}4befed70c1a44813879fce059e88ab310e29b039" -d '{"events": [{"tag": "4c311f56-a73f-47a6-aa58-95f1a774396e", "name": "network-vif-deleted", "server_uuid": "1c4830b5-f1f5-4bfb-bcd9-63c67aed127f"}]}' _http_log_request /opt/mhos/python/lib/python2.7/site-packages/keystoneauth1/session.py:375
neutron/server.log.2:2018-02-01 18:48:54.459 5838 DEBUG neutron.notifiers.nova [-] Nova returned NotFound for event: [{'tag': u'4c311f56-a73f-47a6-aa58-95f1a774396e', 'name': 'network-vif-deleted', 'server_uuid': u'1c4830b5-f1f5-4bfb-bcd9-63c67aed127f'}] send_events /opt/mhos/openstack/neutron/lib/python2.7/site-packages/neutron/notifiers/nova.py:248
neutron/server.log:2018-02-05 06:17:38.871 5840 INFO neutron.wsgi [req-9413e35e-b050-4d88-b8c4-963b6cda07b0 d22bc5405f5841ad817bc6b75dd5b3df ca15303f1f2d4eb8af607f76b10e8250 - default default] 192.168.1.193,192.168.1.177 "GET /v2.0/ports?device_id=1c4830b5-f1f5-4bfb-bcd9-63c67aed127f HTTP/1.1" status: 200 len: 205 time: 0.0538130

[kirandevraaj (kirandevraaj)]

Attaching nova compute logs,

nova boot failed due to error in creating snapshot in ESXi server, as a vmdk by same name already exist in the datastore.

FileAlreadyExistsException: Cannot complete the operation because the file or folder [datastore-nfs] vmware_base/af47c1eb-0285-4fd4-87eb-079539e94726/af47c1eb-0285-4fd4-87eb-079539e94726.1.vmdk already exists.

oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [req-8a323d29-8984-4428-8765-d6d28247739e 27ce7e50022e412d93a9fb6a2f11418a 4da775c085144e818e7aeec9e816b673 - default default] [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] Instance failed to spawn: FileAlreadyExistsException: Cannot complete the operation because the file or folder [datastore-nfs] vmware_base/af47c1eb-0285-4fd4-87eb-079539e94726/af47c1eb-0285-4fd4-87eb-079539e94726.1.vmdk already exists
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] Traceback (most recent call last):
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2236, in _build_resources
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] yield resources
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/compute/manager.py", line 2016, in _build_and_run_instance
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] block_device_info=block_device_info)
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/virt/vmwareapi/driver.py", line 392, in spawn
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] admin_password, network_info, block_device_info)
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] File "/opt/mhos/openstack/nova/lib/python2.7/site-packages/nova/virt/vmwareapi/vmops.py", line 773, in spawn
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] self._use_disk_image_as_linked_clone(vm_ref, vi)
oscompute02_ansible_nova_logs/var/log/nova/compute.log.3:2018-08-10 13:38:22.373 23435 ERROR nova.compute.manager [instance: 585c6fb3-a7ea-4d5c-9d09-85b10f34c5dd] File "/op...

[kirandevraaj (kirandevraaj)]

Issue is still seen with Openstack Stein branch.

 Fault: {u'message': u'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 6b7f6d84-0596-44a2-82d1-ac3ff7383858. Last exception: Cannot complete the operation because the file or folder [nfs-blr-tb01] vmware_base/534a8f6d-54f1-4985-8fb5-5d9fae36c', u'code': 500, u'created': u'2019-11-16T11:16:44Z'}