qemu-img convert does not work with vdi files

[Steve Si]

qemu-img.exe convert -O raw myfile.vdi zz.raw

reports 'error while writing' both in 0.15.0 and Beta rc2
v0.11.1 works fine on same file. myfile.vdi is 3.9GB made by VirtualBox.

[Stefan Hajnoczi (stefanha)]

On Sun, Aug 14, 2011 at 5:32 PM, Steve Si <email address hidden> wrote:
> Public bug reported:
>
> qemu-img.exe convert -O raw  myfile.vdi zz.raw
>
> reports 'error while writing' both in 0.15.0 and Beta rc2
> v0.11.1 works file on same file. myfile.vdi is 3.9GB made by VirtualBox.

Are you sure that there is enough space on the file system to write
out the converted raw image?

Please try qemu-img on a Linux host with the same file. This way we
can be sure it's a generic bug rather than a Windows or
environment-specific problem.

Stefan

[Steve Si (steve2926)]

Please see http://virtuallyfun.superglobalmegacorp.com/?p=1214#comment-967

confirmed bug replicated by neozeed. Seems to be something to do with vdi's that contain 3-4GB of data.
An unpartitioned freshly created 4Gb vdi can be converted to raw OK, but an 8Gb dynamic vdi with 3.8GB of data fails.
Presumably the raw file has sparse data in?? If it has 3+GB of real data then 0.15.0 errors but 0.11.1 does not??

[Stefan Hajnoczi (stefanha)]

On Mon, Aug 15, 2011 at 11:59 PM, Steve Si <email address hidden> wrote:
> Please see
> http://virtuallyfun.superglobalmegacorp.com/?p=1214#comment-967
>
> confirmed bug replicated by neozeed. Seems to be something to do with vdi's that contain 3-4GB of data.
> An unpartitioned freshly created 4Gb vdi can be converted to raw OK, but an 8Gb dynamic vdi with 3.8GB of data fails.
> Presumably the raw file has sparse data in?? If it has 3+GB of real data then 0.15.0 errors but 0.11.1 does not??

I created a 5 GB vdi file. Inside it I created an ext4 file system
and a single 4 GB file with random data on that filesystem.

$ ./qemu-img info test.vdi
image: test.vdi
file format: vdi
virtual size: 5.0G (5368709120 bytes)
disk size: 4.2G
cluster_size: 1048576

The image was converted without error on qemu.git/master running on
Linux amd64. So unfortunately I have not been able to reproduce this.

You could track down the commit which broke this using git-bisect(1).
I recommend first testing QEMU 0.14.0 to see if the bug is present
there.

The steps are:

$ git bisect start master v0.14.0 # broken on master, working on v0.14.0

Then:

$ ./configure [...] && make && qemu-img.exe [...] # check whether it errors

If qemu-img.exe completes successfully:
$ git bisect good
otherwise:
$ git bisect bad

Keep repeating this and you should end up at the commit that
introduced the bug. If you post that it should be relatively easy to
figure out what the cause of the bug is.

Stefan

[Steve Si (steve2926)]

I got binaries from http://homepage3.nifty.com/takeda-toshiya/

tested under Win 7 64-bit as User=Administrator

qemu10.6 - works makes 16/08/2011 18:43 3,096,481,792 test.raw
qemu11.1 - works makes 16/08/2011 18:43 3,096,481,792 test.raw
qemu12.1 - crashes !
qemu12.2 - crashes !
qemu12.5 - error while writing
qemu13.0 - error while writing

Does this help?

[Stefan Hajnoczi (stefanha)]

On Tue, Aug 16, 2011 at 7:07 PM, Steve Si <email address hidden> wrote:
> I got binaries from http://homepage3.nifty.com/takeda-toshiya/
>
> tested under Win 7 64-bit as User=Administrator
>
> qemu10.6 - works  makes      16/08/2011  18:43     3,096,481,792 test.raw
> qemu11.1 - works  makes      16/08/2011  18:43     3,096,481,792 test.raw
> qemu12.1 - crashes !
> qemu12.2 - crashes !
> qemu12.5 - error while writing

I think this crash->error transition is explained by:

commit 8cef921d18025b4ce4e749eed391308033fee232
Author: TeLeMan <email address hidden>
Date: Mon Feb 8 16:20:00 2010 +0800

    qemu-img: use the heap instead of the huge stack array for win32

But we still don't know why you are getting the write error. It would
be fairly easy to find that out by building QEMU from source and
adding fprintf() where the write fails - print out the error code and
sector number.

Stefan

[Stefan Hajnoczi (stefanha)]

On Wed, Aug 17, 2011 at 5:29 PM, Stefan Hajnoczi <email address hidden> wrote:
> On Tue, Aug 16, 2011 at 7:07 PM, Steve Si <email address hidden> wrote:
>> I got binaries from http://homepage3.nifty.com/takeda-toshiya/
>>
>> tested under Win 7 64-bit as User=Administrator
>>
>> qemu10.6 - works  makes      16/08/2011  18:43     3,096,481,792 test.raw
>> qemu11.1 - works  makes      16/08/2011  18:43     3,096,481,792 test.raw
>> qemu12.1 - crashes !
>> qemu12.2 - crashes !
>> qemu12.5 - error while writing
>
> I think this crash->error transition is explained by:
>
> commit 8cef921d18025b4ce4e749eed391308033fee232
> Author: TeLeMan <email address hidden>
> Date:   Mon Feb 8 16:20:00 2010 +0800
>
>    qemu-img: use the heap instead of the huge stack array for win32
>
> But we still don't know why you are getting the write error.  It would
> be fairly easy to find that out by building QEMU from source and
> adding fprintf() where the write fails - print out the error code and
> sector number.

Here is a patch to print more information when the error occurs:
http://patchwork.ozlabs.org/patch/110338/

It applies onto qemu.git/master:
$ cd qemu
$ git am path/to/patch
$ ./configure [...] && make

Stefan

[Steve Si (steve2926)]

What URL can I use to do an svn checkout under MinGW?

[Steve Si (steve2926)]

I downloaded source and ran configure under MinGW but got
Python not found. Use --python=/path/to/python
How to fix pls?

[Stefan Hajnoczi (stefanha)]

On Wed, Aug 17, 2011 at 8:05 PM, Steve Si <email address hidden> wrote:
> I downloaded source and ran configure under MinGW but got
> Python not found. Use --python=/path/to/python
> How to fix pls?

$ ./configure --disable-guest-agent
$ make

Stefan

[Steve Si (steve2926)]

Sorry to be a pain, but now I get

$ configure --disable-guest-agent

Error: zlib check failed
Make sure to have the zlib libs and headers installed.

[Stefan Weil (ubuntu-weilnetz)]

A patch is available here:
http://patchwork.ozlabs.org/patch/110762/

[Steve Si (steve2926)]

If someone could give me a link to a compiled binary, I will test this - thanks.

[Steve Si (steve2926)]

I have applied the patch now and can confirm it works. Would this bug also affect qemu.exe for x86_64?

[Stefan Weil (ubuntu-weilnetz)]

The patch fixes qemu_ftruncate64 (ftruncate) which is also called from
qemu.exe, qemu-system-x86_64.exe and other executables, so the bug
affects them all.

[Aaron Bouzek (nakarti)]

Did this bug come back, or is "error while reading" unrelated but similar?

Oh look, Debian is using an old version, what are the odds?

Distributor ID: Debian
Description: Debian GNU/Linux 6.0.6 (squeeze)
Release: 6.0.6
Codename: squeeze
root@nakvm:~# qemu-img --help
qemu-img version 0.12.5, Copyright (c) 2004-2008 Fabrice Bellard
usage: qemu-img command [command options]
QEMU disk image utility

So now I get to figure out where to get a new copy...

[Aaron Bouzek (nakarti)]

Or an old copy because Ubuntu proffers 1.0 which works fine on the same image.