Corruption of File Data/Filename w/ Virtual VFAT (VVFAT)

[Porcelain Mouse]

Binary package hint: kvm

KVM Version: 1:72+dfsg-1ubuntu6
QEMU Version: 0.9.1-5ubuntu3
Description: Ubuntu 8.10
Release: 8.10

File Data and Filename corruption occur when copying large files to a Virtual FAT (VFAT) drive. I noticed this when copying ISO images between 150MB and 200MB from WindowsXP guest to the Linux host. These ISO images would not boot. Suspecting corruption, I confirmed MD5 hash values of guest and host copies of these ISO images were different. (The guest OS calculated the correct hashes for files on the VFAT drive, but the files available to the host were corrupted.)

Also, filename corruption also occurred. For example, the VFAT directory on the host contained the file "pebuilder3110a.zip". After "exporting" files named "pebuilder.iso", "pebuilder.log", and "pebuilder.log.iso.md5sum.txt" from the guest OS to the Linux host, the VFAT directory contained filenames "pebuilder.iso86.zip" (having the size of the ISO image), and "pebuilder.logmd5sum.txt" (having the size of the pebuilder.log" file). Again, this corruption was not detectable from the guest OS.

Finally, I mounted the guest OS disk image using Linux loop device. Copying the ISO files from the NTFS file system using this method was successful. Using PuTTY's sftp also worked.

To be clear, this corruption was observed when using both KVM and QEMU.

I hope this report is helpful. I can provided more information if it would be valuable. But, this was frustratingly easy to reproduce. I am willing to help in any way.

[Porcelain Mouse (pmouse)]

I forgot to mention this corruption was observed with both QEMU and KVM.

[Dustin Kirkland  (kirkland)]

Thanks for the report.

I'm marking this 'critical' since the reporter claims data corruption. I haven't confirmed it yet.

A couple of questions.
 1) Can you reproduce this on an Ubuntu Jaunty host?
 2) Does the problem only happen when copying data guest->host? What about host->guest?
 3) What protocol are you using to do the copy? scp? ftp? rsync? samba? nfs? Something else?

:-Dustin

[Dustin Kirkland  (kirkland)]

Also, what type of guest disk? qcow? qcow2?

:-Dustin

[Dustin Kirkland  (kirkland)]

And what's the command line you're using to launch this?

[Porcelain Mouse (pmouse)]

Hi Dustin,

    Excellent questions. I should have included that information.

1) Can you reproduce this on an Ubuntu Jaunty host?
     I have not tried Jaunty 9.04 beta. Sorry. What version of KVM and QEMU are provided? I checked the main KVM changelog and I don't see anything about it. I thought I read something about a rewrite of the VFAT code but I cannot find that in the Changelog for the life of me. Off topic, is there a way to peek at other APT repositories to, say, get the Changelog of a package in an upcoming release?
    If you have good reason to believe this is likely to be resolved in Jaunty, let me know. I'll try to move up my upgrade to 9.04 and test for you. My test machine doesn't have the horsepower/RAM for VMs.

2) Does the problem only happen when copying data guest->host? What about host->guest?
    Yes. I have only detected corruption when exporting data from the guest system to the host (i.e. writes). I have read many small files without any trouble and more convenient than Samba...
    Well, you'll be interested in this new behavior. I thought I would just test a big file for you really quick. So, I put a new 150MB ISO image in the VFAT directory and started KVM. But, this time, I received this error:

    kvm: block-vvfat.c:96: array_get: Assertion `index < array->next' failed.

I created a new directory, moved the ISO image there, and updated the -drive parameter to point to the new directory. KVM started correctly. I performed the test copy, importing the ISO image file without corruption. I believe this problem is isolated to large file exports (i.e. VFAT writes).
    Now, I'm lost regarding that error. That sounds like the directory file on the host is getting corrupted, in addition to filenames and file data. But, I don't see any bogus data in the directory file, although I don't really know what or how to look for that. The other option is that there is some restriction on the Virtual FAT device that I don't understand. For example, the last time I used this particular VM, WindowsXP was constantly annoying me about the VFAT drive being full and that I should run Disk Cleanup on it. WindowsXP sees the VFAT drive as having a total capacity of 512MB and there was, at that time, more than 512MB used by files in the VFAT directory and its subdirectories on the host. However, the drive was nearly empty when I first noticed this problem and my tests were completed before the drive was "full." It filled up as I kept different versions of the same ISO file for comparison.

3) What protocol are you using to do the copy? scp? ftp? rsync? samba? nfs? Something else?
    The copying that produced the corruption was Windows Explorer copy/paste or drag/drop. I'm sorry I didn't try copy. SFTP network-based export worked perfectly. Mounting the NTFS filesystem within the VM image (after converting to raw with qemu-img convert) and using cp from the host also worked. I assume all TCP/IP or Network based transfers are safe.

4) Also, what type of guest disk? qcow? qcow2?
    Originally, I was using qcow2 format. But, before I thought of SFTP (doh!) I decided to try to loopback mount the VM disk image, w...

[Dustin Kirkland  (kirkland)]

I think this might be fixed by:
 * http://git.kernel.org/?p=virt/kvm/kvm-userspace.git;a=commit;h=385aae33c08e17ec63671b9fa1c503c21283bc1f

:-Dustin

[Dustin Kirkland  (kirkland)]

Soren disagrees in IRC though...

[Dustin Kirkland  (kirkland)]

Soren indicates that VVFAT is a very non-standard way of doing disk allocation for VM's.

Is there some reason that you're using this rather than any of the more standard disk image formats?

:-Dustin

[Porcelain Mouse (pmouse)]

Granted. I'm sure that's true, but yes, I am using VVFAT for two reasons: (1) it is direct, read/write, host-to-guest file access, and (2) it is the easiest method to transfer files between host and guest.

When I first encountered the need to transfer files I had on my host to my VM, I looked for direct methods because it seemed more strait forward. VVFAT worked for small files, which is all I had to move for a long time. Also, I didn't test writes to the VVFAT because I didn't do that much, until recently. After using this feature for a while, I found it to be extremely convenient and useful.

Secondly, I found VVFAT much easier to use than the network-based transfer methods. I didn't want to use QEMU's CIFS/SMB/Samba option because it does not allow the use of an alternate Samba configuration file. I didn't want to have to configure /etc/smb.conf just run my VM in user space. (At least, I couldn't figure out how to do it, otherwise.) Similar inconveniences occur using FTP and SSH (which I'm using now, but wouldn't want to at work).

Generally speaking, I'm not really using VVFAT as a disk image. I'm using it as a file transfer method. Using an actual disk image would require mounting the disk image in on the host, either via root access, FUSE, or using filesystem editing tools, which are inconvenient. I have created ISO9660 images to import files, which works, but requires creating a new image every time I need to import more files and doesn't support writes, of course. (I don't know how UDF would work on a disk image file. Would that work?)

I suppose the next thing I should try is an NTFS-3g filesystem which I could mount with FUSE in my Linux host and natively in my Windows VM. That's still less convenient than VVFAT, however, since I would have to mount/umount the image on the host to get access and the VM would still have to be off. I would still prefer VVFAT to all of these options.

I really appreciate your help. Thank you for taking my report seriously. If VVFAT is buggy and cannot be fixed easily, I defer to the judgment of others and ask only that this limitation be well documented in the primary references. But, I'm convinced that a direct, host-to-guest file access scheme is valuable. If there is another solution, I apologize for my ignorance and I would appreciate being pointed in the right direction.

[Anthony Liguori (anthony-codemonkey)]

vvfat is not commonly used/tested in upstream KVM or QEMU. Patches are certainly welcome but it's definitely considered an experimental feature.

[Dustin Kirkland  (kirkland)]

Although I haven't personally confirmed this, I'm going to mark it 'confirmed', since vvfat support is known to be 'experimental' in upstream QEMU. Furthermore, I'm going to reduce the importance from critical to wishlist. As the upstream maintainer has stated, this feature is known to be experiment, it's not under active development, though patches are welcome.

:-Dustin