vmhgfs-fuse inconsistencies cause Git index corruption and Bus Error crashes; older vmhgfs was fine

[Jamie Lokier]

Summary: The switch from vmhgfs to newer vmhgfs-fuse (in open-vm-tools) results in data corruption and bus errors, due to particularly bad cache coherency issues. I noticed this when using Git.

Detail:

I'm running Ubuntu 16.04 in a VM on my Macbook using VMware Fusion 8.0.

I use VMware Fusion's folder sharing to access the Mac's home directory from Linux.
From the Linux point of view, this is vmhgfs.

Previously I was running an older version of Ubuntu, and I upgraded it recently.

Following the upgrade, I found the vmhgfs kernel module was no longer available, vmhgfs-fuse was packaged as part of open-vm-tools, and the syntax needed in /etc/fstab changed significantly. This was inconvenient at the time, but not a serious problem.

What has emerged as a serious problem, though, is when I run almost any Git command on one of these shared folders, I get one or both of these errors:

    - Reports of Git index corruption
    - Git crashes with a Bus Error

Investigations found this test case:

     $ echo foo > TEST; ls -l TEST; echo longer > TEST.new; mv TEST.new TEST; ls -l TEST; sleep 6; ls -l TEST
     -rw-rw-r-- 1 jamie jamie 4 May 7 03:26 TEST
     -rw-rw-r-- 1 jamie jamie 4 May 7 03:26 TEST <-- This should have size 7, not 4.
     -rw-rw-r-- 1 jamie jamie 7 May 7 03:26 TEST <-- After the 6 second delay it shows the correct size.

That's awful behaviour for a filesystem, and it makes vmhgs-fuse quite useless for serious interactions in which we write to the filesystem.

The Bus Errors which Git had were due to it calling mmap() with the reported size, and then finding that some of the mapped range was not readable because it wasn't the real size.

I tried disabling caching with the documented FUSE options:

    - entry_timeout=0
    - attr_timeout=0

Unfortunately vmhgfs-fuse ignores these. It has its own /hard-coded/ 5 second cache for attributes, separate from FUSE's cache, and it doesn't even notice when it itself is responsible for changing a file on the client size. The fact you can't disable the first and the second doesn't work are individually both quite bad.

The source for vmhgfs-fuse defines the hard-coded timeout as CACHE_TIMEOUT in cache.c.

Since the old vmhgfs worked, is there a way to install it on Ubuntu 16.04?

(This has forced me to stop using vmhgfs-fuse and look for alternatives to access my Mac's home directory from the VM. Unfortunately NFS has user id mapping problems (changing either user id isn't a solution for me), Mac SMB can't convey executability to Linux CIFS (this also breaks Git), and Samba smbclient currently has a bug whereby it can't connect to Mac SMB shares with a password! So I am still looking for a solution! :)

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Robert C Jennings (rcj)]

Jamie, I have copied this bug upstream for VMWare to look at in https://github.com/vmware/open-vm-tools/issues/90. And I have added Ravindra and Oliver for feedback here.

[Robert C Jennings (rcj)]

Jamie et al,

from the upstream developer in the git issue linked in comment #2:
Hi there,
I am the developer for this Shared Folders feature.

Thanks so much for reporting this. I am not sure how we missed this.
I will file a bug and get this issue addressed.

It should be treated in exactly the same manner as the kernel client did, which it clearly is not.

Steve

[Robert C Jennings (rcj)]

When updates for this are ready we will work to get them into Ubuntu

[David D Miller (justdave)]

fwiw, the old vmhgfs module does appear to work (obviously not a long-term workaround). You can build and install the one distributed with open-vm-tools-dkms like so:

cd /usr/src/open-vm-tools-10.0.7/vmhgfs
make
cp vmhgfs.ko /lib/modules/`uname -r`/updates/dkms/
depmod
insmod vmhgfs.ko

YMMV given that VMWare doesn't recommend using it with the newer kernels.

[[e-mail] (lousybrit)]

Hi there Robert et al,
I am the developer for this Shared Folders feature. (Robert will know this from his other post and emails.)

Thanks so much for reporting this. I am not sure how we missed this.
I will file a bug and get this issue addressed.

It should be treated in exactly the same manner as the kernel client did, which it clearly is not.

I have now fixed this issue (at least internally to VMware) and I hope it will not be too long before it makes it into the public domain.

Steve

[Jamie Lokier (jamie-shareable)]

I am using this patch to fix the specific problem with rename()
which looks like a clear bug no matter what the cache behaviour.
In trivial tests, Git has started behaving after this. I haven't
stress tested it to see if other cache consistency issues still
cause problems.

This patch doesn't fix cache consistency in general, and it doesn't
address other changes from vmhgfs such as less control over file
permissions than before.

If it's going to use a fixed cache timeouts without checking,
it should probably do what NFS does: close-to-open consistency
and unlock-to-lock consistency, which applications have
historically been ok with.

--- open-vm-tools-10.0.7-3227872/vmhgfs-fuse/main.c~ 2016-02-16 20:06:49.000000000 +0000
+++ open-vm-tools-10.0.7-3227872/vmhgfs-fuse/main.c 2016-05-09 05:56:59.423936135 +0100
@@ -622,6 +622,7 @@ hgfs_rename(const char *from, //IN: fro

    res = HgfsRename(absfrom, absto);
    HgfsInvalidateAttrCache(absfrom);
+ HgfsInvalidateAttrCache(absto);

 exit:
    LOG(4, ("Exit(%d)\n", res));