VMware FCD driver failed to lock the file

[Avishay Traeger]

At times, creating a clone or creating a volume from snapshot fails with "failed to lock file".

2019-10-23 14:33:05,442.442 34 INFO cinder.volume.flows.manager.create_volume [req-4535930f-4d94-48ab-9249-626258ea1398 - 0 - - -] Volume dae99b78-0fd9-4968-9fb7-43e3e4fdfc11: being created as snap with specification: {'status': u'creating', 'volume_size': 50, 'volume_name': 'volume-dae99b78-0fd9-4968-9fb7-43e3e4fdfc11', 'snapshot_id': 'fb75e1e1-abed-4f8d-808b-ab6ff60d0106'}
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall [-] in fixed duration looping call
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall Traceback (most recent call last):
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall File "/usr/lib/python2.7/site-packages/oslo_vmware/common/loopingcall.py", line 75, in _inner
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall self.f(*self.args, **self.kw)
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall File "/usr/lib/python2.7/site-packages/oslo_vmware/api.py", line 452, in _poll_task
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall raise task_ex
2019-10-23 14:33:31,522.522 34 ERROR oslo_vmware.common.loopingcall VimFaultException: A general system error occurred: Failed to lock the file: api = DiskLib_Open, snapshotPath.c_str() = /vmfs/volumes/5d8ccaf8-d9eaeda1-f3da-e41d2d00d270/fcd/0cf3539c95134c91858f69f6ac237f0f.vmdk

[Avishay Traeger (avishay-il)]

In addition, if I log into one of the ESXs and run /bin/vmfsfilelockinfo, it says:
"0cf3539c95134c91858f69f6ac237f0f.vmdk" is not locked by any ESX host and is Free

If I continue to retry it fails in the same way. It doesn't seem related to the size of the source or the amount of data in it.

[Avishay Traeger (avishay-il)]

Hello Vipin, have you had a chance to look into this issue?

[Vipin Balachandran (vbala)]

Avishay, This intermittent lock failure is a vSphere bug and the fcd team is aware of it. I don't think we can have a workaround in the Cinder driver.