VMware driver: unused cached images never clear from datastore's cache folder (usually named vmware_base)

[dan wendlandt]

customer reports that that even when they no longer have a VM in their vCenter inventory using a particular image, they are unable to delete the vmdk corrsponding to that image in the vmware_base directory.

Note, the true problem here is that if one cycles through a lot of different images in your deployment, these images are not cleared from the datastore "cache" in any automatic fashion. This issue was encountered when someone tried to perform this task manually using vCenter client. They got the error: “cannot delete file <filename>”

[Shawn Hartsock (hartsock)]

I've seen this behavior.

The manual work-around is to use one of the vSphere tools (CLI, vSphere Client, NGC) to manually delete the extra images in the cache. This is not a good long-term solution and causes problems with linked_clone based images should an administrator delete the wrong file from cache.

* be careful in the fix not to remove the wrong item from cache
* be mindful of linked_clone based instances which will be partially linked to the file in cache (these instances will crash if a cache file is deleted).

[dan wendlandt (danwent)]

Hi Shawn,

The original comment says:

This issue was encountered when someone tried to perform this task manually using vCenter client. They got the error: “cannot delete file <filename>”

So I believe the customer already tried your workaround and says it does not work. The bug is not looking to automatically delete the images, it just mentions that as the explanation for why the user is trying to delete those images.

[Shawn Hartsock (hartsock)]

Thanks for the clarification.

I've changed the bug title to reflect the precise issue. When all the instances using an image in the cache (happens to be named vmware_base by default) we'll have to figure out why when the count of images using the cache hits zero a delete file isn't being triggered.

[Shawn Hartsock (hartsock)]

I have managed to reproduce this bug. You can remove the VMDK files by enabling and using SSH to that particular host, then you can use 'rm -rf' to remove the files from the file system manually. I'm still looking into this.

[dan wendlandt (danwent)]

Thanks Shawn. Is it any ESX host with that datastore that you can log into, or a specific one?

[Shawn Hartsock (hartsock)]

This appears to be related to the "use_linked_clone" configuration and some code that creates a hidden file. I'm still not certain where or when the "hidden" linked clone file shows up. It does not seem to matter which datastore is used.

[Gary Kotton (garyk)]

This will be dealt with by https://blueprints.launchpad.net/nova/+spec/vmware-image-cache-management

[Gary Kotton (garyk)]

Treated by https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/vmware-image-cache-management,n,z

[Matt Riedemann (mriedem)]

Removing from rc1 since this is tied to blueprint vmware-image-cache-management which didn't get a feature freeze exception for icehouse.

[Gary Kotton (garyk)]

Matt, the vmware-image-cache-mamagement was approved an implemented.

[Gary Kotton (garyk)]

This was approved towards the end of icehouse - https://review.openstack.org/#/c/56416/