Qemu-img -f raw causes a black hole

[Githlar]

When the program qemu-img with the options of -f raw is used it has some very strange consequences.

Steps to reproduce:
1. mkdir testdir
2. cd testdir
3. qemu-img create -f raw test.img 20G (Other sizes also work)
4. Program starts writing the file and becomes un-killable
5. Open a new terminal in the same directory
6. ls -lah shows that the size of the data in the folder increases up to the point specified in qemu-img, however test.img retains a size of zero. This is why I'm referring to it as a "black hole" If that data isn't getting written to the file, where is it going?

The last time I did this (forgetting about the side effect), I believe it ended up corrupting many of my existing files in my home directory and possibly elsewhere. After doing this I was no longer able to use my wireless connection (or even manage anything with Network Manager - everything was greyed out) and the Suspend/Power Off menu in Gnome Shell disappeared. I'm sure more damage was done, I have just yet to find it. I ended up having to re-install to fix everything (especially to use my wireless connection).

What's odd about this is that qemu-img takes under a second to create a 20G file. My analysis of this coming from a programming background is that the program opens a file handle, moves the filepos to 20,000,000,000 and then closes the file - effectively making it a 20GB file. But, for some reason eCryptfs doesn't play nicely with this file operation. When used in another filesystem (ext* or vfat are the ones I've tried) there are no problems whatsoever. However, if you even attempt to move these files into the eCryptfs it exibits the same aforementioned behavior.

I don't believe this to be the fault of Qemu (though I'm still attempting to look through the code to see exactly how it does the operation) because I've never had any problems with it unless it was used with an eCryptfs filesystem.

As far as qemu-img hanging: it will finish writing data, however it will remain as an uninterruptable process. And while qemu-img is in this state, any programs that attempt to touch the file (mv, rm, etc.) will also hang in that un-killable state.

I don't know what I should put up for diagnostics, nor do I have anything left due to the reformat and reinstall. But, if I can provide any information, I will gladly attempt to re-create the situation (albeit in a virtual machine so that I trash my installation again).

As I'm not sure if this overwrote system files or not, I'm not going to flag it as a security vulnerability - though it very well may be.

[Githlar (githlar-deactivatedaccount)]

Just as a clarification, qemu-img becomes unkillable once it is started. The only way to kill it is to restart. All the while it's writing data somewhere to the disk. Even once it is finished, qemu-img hangs in this state and never finishes.

[Tyler Hicks (tyhicks)]

On 2012-02-20 04:26:56, Githlar wrote:

Hello!

> 6. ls -lah shows that the size of the data in the folder increases up to the point specified in qemu-img, however test.img retains a size of zero. This is why I'm referring to it as a "black hole" If that data isn't getting written to the file, where is it going?

Are you running a 32 bit kernel?

> What's odd about this is that qemu-img takes under a second to create a
> 20G file. My analysis of this coming from a programming background is
> that the program opens a file handle, moves the filepos to
> 20,000,000,000 and then closes the file - effectively making it a 20GB
> file. But, for some reason eCryptfs doesn't play nicely with this file
> operation. When used in another filesystem (ext* or vfat are the ones
> I've tried) there are no problems whatsoever. However, if you even
> attempt to move these files into the eCryptfs it exibits the same
> aforementioned behavior.

eCryptfs doesn't have sparse file support. It has to encrypt zeros and
write the ciphertext out to the lower filesystem. This has some security
reasoning behind it (to not give information about where holes in the
file are) and also because eCryptfs doesn't have a good way of
identifying where holes in the lower file are at.

[Githlar (githlar-deactivatedaccount)]

I am running a 64-bit kernel, however the same bug applies to 32-bit kernels as well. I only recently upgraded to 64-bit and had used Qemu on 32-bit for years and had the same problem. Up to this point though, I didn't realize what it was doing.

I can understand why there is no sparse file support, but wouldn't the filesystem driver determine how Qemu allocates a file? As an example, FAT32 does not support sparse files either, however a Qemu image creation on this filesystem has no issues (granted, it's not as quick as making an image on ext filesystems as it has to write out all the zeros). That's I made this a bug report.

But, when creating a large file in a FAT32 filesystem I can use ls -lah and see both the size of the folder and the size of the file match - as in the file has a non-zero file size.

[Githlar (githlar-deactivatedaccount)]

After doing so more thorough testing in a virtual machine, it seems some of my prior information is incorrect. Once qemu-img finishes writing the file, it does exit normally and the file size is set correctly. Though I still think it's an issue that potentially corrupts existing files.

Now, I could be wrong, but I didn't do much other than that when my last installation gave out. It exhibited some very weird behavior. Like I said, I couldn't connect to my Wireless connection and the Networking option in Gnome was completely disabled with no way to enable it. Then there was the missing Suspend/Shutdown menu... Some really odd stuff.

It's also strange that a program writing to a file in eCryptfs becomes unkillable - as does any program that touches the file during the operation.