data loss in conversion to or from vpc

[Scott Moser]

Given the script script, I would expect that out.raw* files would
be bit for bit identical. And indeed, a little test image I made, they were.

However, when I tried it on a oneiric compressed qcow disk, they were not.
With 'in' of oneiric-server-cloudimg-amd64-disk1.img, the
output looks like:

f4a076d62bf13a5ba0c9c82496fe3303 /tmp/convtest.vpc
c1f6e09e96361c4f56aea0b9ea845a36 /tmp/convtest.raw
c1f6e09e96361c4f56aea0b9ea845a36 /tmp/convtest.raw.from-file
0f4b00a770aa795b71de4ca3311898fa /tmp/convtest.raw.from-vpc
-rw-r--r-- 1 2147483648 2011-10-26 14:31 /tmp/convtest.raw
-rw-r--r-- 1 2147483648 2011-10-26 14:31 /tmp/convtest.raw.from-file
-rw-r--r-- 1 2147991552 2011-10-26 14:31 /tmp/convtest.raw.from-vpc
-rw-r--r-- 1 709017088 2011-10-26 14:30 /tmp/convtest.vpc

Note, that the from-vpc file is ~ 500k larger than the raw files
and obvisously the md5sum differs as a result.

--
#!/bin/sh
in=${1}; out=${2}
qemu-img convert -O vpc /dev/stdin $out.vpc < "$in"
qemu-img convert -O raw /dev/stdin $out.raw < "$in"
qemu-img convert -O raw "$in" "$out.raw.from-file"
qemu-img convert -f vpc -O raw $out.vpc $out.raw.from-vpc
md5sum $out.vpc $out.raw $out.raw.from-file $out.raw.from-vpc
ls -l $out.vpc $out.raw $out.raw.from-file $out.raw.from-vpc

ProblemType: Bug
DistroRelease: Ubuntu 11.10
Package: qemu-kvm 0.14.1+noroms-0ubuntu6
ProcVersionSignature: Ubuntu 3.0.0-12.20-generic 3.0.4
Uname: Linux 3.0.0-12-generic x86_64
ApportVersion: 1.23-0ubuntu3
Architecture: amd64
Date: Wed Oct 26 13:53:33 2011
EcryptfsInUse: Yes
InstallationMedia: Ubuntu 11.10 "Oneiric Ocelot" - Release amd64 (20111012)
KvmCmdLine: Error: command ['ps', '-C', 'kvm', '-F'] failed with exit code 1: UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
MachineType: LENOVO 7417CTO
PccardctlIdent:
 Socket 0:
   no product info available
PccardctlStatus:
 Socket 0:
   no card
ProcEnviron:
 PATH=(custom, user)
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-3.0.0-12-generic root=UUID=f9832678-e9fb-41c5-8edb-5edd5200ed0a ro quiet splash vt.handoff=7
SourcePackage: qemu-kvm
UpgradeStatus: No upgrade log present (probably fresh install)
dmi.bios.date: 12/06/2010
dmi.bios.vendor: LENOVO
dmi.bios.version: 7UET91WW (3.21 )
dmi.board.name: 7417CTO
dmi.board.vendor: LENOVO
dmi.board.version: Not Available
dmi.chassis.asset.tag: No Asset Information
dmi.chassis.type: 10
dmi.chassis.vendor: LENOVO
dmi.chassis.version: Not Available
dmi.modalias: dmi:bvnLENOVO:bvr7UET91WW(3.21):bd12/06/2010:svnLENOVO:pn7417CTO:pvrThinkPadT400:rvnLENOVO:rn7417CTO:rvrNotAvailable:cvnLENOVO:ct10:cvrNotAvailable:
dmi.product.name: 7417CTO
dmi.product.version: ThinkPad T400
dmi.sys.vendor: LENOVO

[Daniel (daniel-redhat-bugs)]

Description of problem:
If you create a raw file, convert it to VPC format with qemu-img, and then convert it back to raw format, the resulting raw file is not the same as the original. All the other disk formats roundtrip reliably, but not VPC.

Version-Release number of selected component (if applicable):

How reproducible:
qemu-0.11.0-6.fc11.i586

Steps to Reproduce:
1. dd if=/dev/zero of=input.img bs=1M count=50
2. qemu-img convert -f raw -O vpc input.img output.img
3. qemu-img convert -f vpc -O raw output.img input2.img
4. md5sum input.img
5. md5sum input2.img
6. ls -l input.img
7. ls -l input2.img

Actual results:
25e317773f308e446cc84c503a6d1f85 input.img
b699e19dd8d468dbd7264df67ca7f798 input2.img
-rw-rw-r--. 1 berrange berrange 52428800 2009-10-13 11:44 input.img
-rw-r--r--. 1 berrange berrange 52432896 2009-10-13 11:45 input2.img

Expected results:
input.img and input2.img should be identical in everyway.

Additional info:

[Daniel (daniel-redhat-bugs)]

Created attachment 364572
Example failure when testing libvirt cloning API

FYI, the original way we discovered this bug was via the libvirt TCK's storage cloning tests. That tests creates a raw volume with a special data pattern, and then tests conversion to & from every QEMU writable disk format. We currently have to mark the VPC tests as TODO, since we get md5sum mis-matches after VPC -> raw conversion with unexpected extra NULL bytes appended on the final raw file.

[Mark (mark-redhat-bugs)]

kevin: sounds like something you could sort out quickly if you had some time to spare :-)

[Kevin (kevin-redhat-bugs)]

This is known problem, but I think this is not a qemu bug but a shortcoming of the VHD format (or VirtualPC's handling of that format) where we are doomed to do it wrong at some place. Adding zeros seemed to be the least wrong thing when I implemented it.

For reference, I explained the behaviour in http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00570.html:
> I think this depends on the definition of "correct". VPC unfortunately
> seems to like contradicting values in its image - CHS geometry and image
> size rarely match. The guest seemed to see the CHS geometry, so I
> changed the code to take the geometry instead of the size header field
> to calculate the image size.

If we didn't round up to the next matching CHS values and used the exact value, our images would seem to be truncated in VirtualPC. You shouldn't observe the problem when you choose the image size so that its size can be represented exactly in a CHS geometry.

[Daniel (daniel-redhat-bugs)]

Worked around the patch in this commit, by adding 4k to the disk size to make it a nice size for CHS geom

http://libvirt.org/git/?p=libvirt-tck.git;a=commit;h=9eedd1ed9887fe294c8d856fc7142c6716799216

[Scott Moser (smoser)]

Just to test whether it was reading or writing that was the error, I tried this:
 $ qemu-img info /tmp/convtest.vpc
image: /tmp/convtest.vpc
file format: vpc
virtual size: 2.0G (2147991552 bytes)
disk size: 645M

The size there is incorrect, and identical to what the size of the target is after converting back to raw.

Then, testing vboxmanage:
 $ vboxmanage convertfromraw /tmp/convtest.raw /tmp/convtest.vpc.vboxmanage
 $ qemu-img info /tmp/convtest.vpc.vboxmanage | grep "irtual size:"
  virtual size: 2.0G (2147483648 bytes)
  $ qemu-img convert -O raw /tmp/convtest.vpc.vboxmanage /tmp/convtest.raw.from-vbox-vpc
  $ ls -l /tmp/convtest.raw.from-vbox-vpc
  -rw-r--r-- 1 smoser smoser 2147483648 2011-10-26 14:58 /tmp/convtest.raw.from-vbox-vpc
  $ md5sum /tmp/convtest.raw.from-vbox-vpc
  c1f6e09e96361c4f56aea0b9ea845a36 /tmp/convtest.raw.from-vbox-vpc

So, it seems there is a bug writing the vpc format, while reading the format as written by vboxmanage works fine.

[Scott Moser (smoser)]

This bug seems at least related : https://bugzilla.redhat.com/show_bug.cgi?id=528678 .

[Serge Hallyn (serge-hallyn)]

Thanks for the report and the buglink, Scott.