qemu-img convert a overlay qcow2 image into a entire image

[chengkang]

I have a base image file "base.qcow2" and a delta qcow2 image file "delta.qcow2" whose backing file is "base.qcow2".

Now I use qemu-img to convert "delta.qcow2" and will get a new image file "new.qcow2" which is entire and equivalent to combination of "base.qcow2" and "delta.qcow2".

In fact,I don't want to get a complete image.I just want to convert delta qcow2 image file "A" to a New delta overlay qcow2 image "B" which is equivalent to "A". So the "new.qcow2" is not what i want. I have to admit that this is not bug. Could you please take this as a new feature and enable qemu-img to convert qcow2 overlay itself?

[Eric Blake (eblake)]

On 02/05/2017 09:19 PM, wayen wrote:
>
> I have a base image file "base.qcow2" and a delta qcow2 image file
> "delta.qcow2" whose backing file is "base.qcow2".
>
> Now I use qemu-img to convert "delta.qcow2" and will get a new image
> + file "new.qcow2" which is entire and equivalent to combination of
> "base.qcow2" and "delta.qcow2".
>
> In fact, i just want to convert the delta qcow2 image file and get a new
> delta overlay qcow2 image file. So the "new.qcow2" is not what i want. I
> have to admit that this is not bug. Could you please take this as a new
> feature and enable qemu-img to convert images in-place?

This feature already exists. Use:

qemu-img rebase -F $backing_fmt -b '' -f qcow2 delta.qcow2

and now delta.qcow2 will be a complete image.

--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[chengkang (chengkang)]

@Eric Blake. Sorry, I didn't make it clear. In fact, I don't want to get a complete image. I just want to convert qcow2 overlay and get a new qcow2 overlay. Maybe you think my intention is meaningless， but this is what I want.

[Thomas Huth (th-huth)]

Can't you simply do a "cp delta.qcow2 new.qcow2" ?

[chengkang (chengkang)]

@Thomas Huth, when we use qemu-img to convert a image "A" and will get a new image "B" which is equivalent to "A" . But "B" may be a lot different from "A",such as file size. The file size of "B" is usually less than "A" and this is very valuable to me. So I can't simply do a "cp delta.qcow2 new.qcow2" ?

[Thomas Huth (th-huth)]

I meant to copy B, not A ... but I likely just haven't really understood what you're really trying to do here, so never mind.

[chengkang (chengkang)]

@Thomas Huth, I just want to reduce the file size of qcow2 overlay image by this conversion.

[Eric Blake (eblake)]

On 02/08/2017 02:16 AM, wayen wrote:
> @Thomas Huth, I just want to reduce the file size of qcow2 overlay image
> by this conversion.

Are you merely trying to sparsify the file (punch holes on the portion
of the file that is not mapped to disk), so that the apparent file size
is unchanged, but the actual disk usage is minimized? That's probably
already possible (virt-sparsify from libguestfs seems to be able to do
it; look at what steps it uses).

Or are you asking for a way to defragment the file, so that the apparent
size shrinks because all occupied clusters are now contiguous, so that
there are no longer any need for holes? The end result is the same
amount of disk usage, but right now, the only way to defragment is to
convert from one image to a copy; we don't support in-place
defragmentation yet. So far, no one has come up with a compelling
reason why it is needed, or a patch to implement it, but there's no
technical reason why it can't be done.

--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[chengkang (chengkang)]

@Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image. As you said, the actual disk usage is minimized and the apparent file size is unchanged.It is very valuable for me, because it means that my disk can hold more files.

But we need to be careful when transfer the sparse qcow2 overlays. Because some tools do not support sparse file and will convert it into a common file,for example, scp. And I doubt what will happen when transfer sparse files between different file systems.

So I want to use qemu-img to convert the sparse qcow2 overlay into common file. If this was feasible, the holes in the sparse overlay will be removed and the above problems will disappear.

[Stefan Hajnoczi (stefanha)]

On Thu, Feb 09, 2017 at 02:05:54AM -0000, wayen wrote:
> @Eric Blake, I used virt-sparsify to sparsify the qcow2 overlay image.
> As you said, the actual disk usage is minimized and the apparent file
> size is unchanged.It is very valuable for me, because it means that my
> disk can hold more files.
>
> But we need to be careful when transfer the sparse qcow2 overlays.
> Because some tools do not support sparse file and will convert it into a
> common file,for example, scp. And I doubt what will happen when transfer
> sparse files between different file systems.
>
> So I want to use qemu-img to convert the sparse qcow2 overlay into
> common file. If this was feasible, the holes in the sparse overlay will
> be removed and the above problems will disappear.

You can use the drive_mirror HMP command or drive-mirror QMP command on
a running QEMU instance to copy the data to a new file and switch to the
new file.

I'm curious what exactly is being optimized by copying out a fresh qcow2
image.

Please post the output of:

  $ stat delta.qcow2
  $ qemu-img map delta.qcow2
  $ stat new.qcow2
  $ qemu-img map new.qcow2

This will allow us to see the size and data layout differences.

Maybe a new command should be added to QEMU to do the optimization
in-place. This would avoid the disk space overhead associated with
drive-mirror, cp, qemu-img convert, etc.

Stefan

[Stefan Hajnoczi (stefanha)]

On Tue, Feb 14, 2017 at 02:17:16AM -0000, wayen wrote:
> ** Attachment added: "qemu-img map new.qcow2 output"
> https://bugs.launchpad.net/qemu/+bug/1662050/+attachment/4818564/+files/qemu_img_map_new_qcow2.txt

Thanks for posting the attachments.

I ran a script to find unallocated clusters in the delta.qcow2 host
file. Most are actually qcow2 metadata (L1/L2 tables, refcount blocks).

This output shows that any image file size reduction you are hoping to
achieve can only come from zero clusters.

There are no holes in the files that would result in significant image
file size reduction if a new image were written out.

Just wanted to share this info in case anyone else is thinking about how
to optimize qcow2 files. I still think rewriting images sequentially
can be useful - if internal snapshots were used and deleted then COW can
result in holes.

Hole at 0 size 5.0 clusters
Hole at 393216 size 1.0 clusters
Hole at 589824 size 1.0 clusters
Hole at 1114112 size 1.0 clusters
Hole at 1310720 size 1.0 clusters
Hole at 1507328 size 1.0 clusters
Hole at 1703936 size 1.0 clusters
Hole at 2293760 size 1.0 clusters
Hole at 2621440 size 1.0 clusters
Hole at 3080192 size 1.0 clusters
Hole at 5111808 size 1.0 clusters
Hole at 6291456 size 1.0 clusters
Hole at 30408704 size 1.0 clusters
Hole at 47906816 size 1.0 clusters
Hole at 142671872 size 1.0 clusters
Hole at 219545600 size 1.0 clusters
Hole at 667090944 size 1.0 clusters
Hole at 853868544 size 1.0 clusters
Hole at 1562640384 size 1.0 clusters
Hole at 2147483648 size 1.0 clusters
Hole at 2617180160 size 1.0 clusters
Hole at 3411148800 size 1.0 clusters
Hole at 4107075584 size 1.0 clusters
Hole at 4294967296 size 1.0 clusters
Hole at 4452646912 size 1.0 clusters
Hole at 4792057856 size 1.0 clusters
Hole at 5494865920 size 1.0 clusters
Hole at 5645271040 size 1.0 clusters
Hole at 5702483968 size 1.0 clusters
Hole at 6187188224 size 1.0 clusters
Hole at 6442450944 size 1.0 clusters
Hole at 6862995456 size 1.0 clusters
Hole at 6987317248 size 1.0 clusters
Hole at 7567245312 size 1.0 clusters
Hole at 8135245824 size 1.0 clusters
Hole at 8590589952 size 1.0 clusters
Hole at 8613462016 size 1.0 clusters
Hole at 9055436800 size 1.0 clusters
Hole at 9703522304 size 1.0 clusters
Hole at 10279321600 size 1.0 clusters
Hole at 10737418240 size 3.0 clusters
Hole at 10844372992 size 1.0 clusters
Hole at 11167858688 size 1.0 clusters
Hole at 11209605120 size 1.0 clusters
Hole at 11209801728 size 1.0 clusters
Hole at 11730944000 size 1.0 clusters
Hole at 12183207936 size 1.0 clusters
Hole at 12705464320 size 1.0 clusters
Hole at 12884901888 size 1.0 clusters
Hole at 13444120576 size 1.0 clusters
Hole at 13910016000 size 1.0 clusters
Hole at 14182711296 size 1.0 clusters
Hole at 15025635328 size 1.0 clusters
Hole at 15032385536 size 1.0 clusters

The following script draws the allocated clusters and holes in the image
file. I took your qemu-img map output, filtered out any lines with
base.qcow2, and sorted using sort -k3 -g to sort on the "Mapped to"
field. Then I ran ./qcow2-map-svg.py <filtered.txt >output.svg.

#!/usr/bin/python3
import sys
import io

COLOR_ALLOCATED = '#ffaaaa'
COLOR_HOLE = '#999999'

def...

[chengkang (chengkang)]

Is there any way to remove holes from sparse qcow2 overlay? It's very important to me. I am looking forward to your reply.

[Lidong Chen (jemmy858585)]

Hi wayen:
    Which version are you used?
    I also find this problem on old version qemu, and i write a patch
for it. It works.
    I'm not sure the mainline version have solve this problem.
    what command are you used?

On Mon, Apr 10, 2017 at 10:14 AM, wayen <email address hidden> wrote:
> Is there any way to remove holes from qcow2 overlay images? It's very
> important to me. I am looking forward to your reply.
>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1662050
>
> Title:
> qemu-img convert a overlay qcow2 image into a entire image
>
> Status in QEMU:
> Incomplete
>
> Bug description:
> I have a base image file "base.qcow2" and a delta qcow2 image file
> "delta.qcow2" whose backing file is "base.qcow2".
>
> Now I use qemu-img to convert "delta.qcow2" and will get a new image
> file "new.qcow2" which is entire and equivalent to combination of
> "base.qcow2" and "delta.qcow2".
>
> In fact,I don't want to get a complete image.I just want to convert
> delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
> which is equivalent to "A". So the "new.qcow2" is not what i want. I
> have to admit that this is not bug. Could you please take this as a
> new feature and enable qemu-img to convert qcow2 overlay itself?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions
>

[chengkang (chengkang)]

Hi Lidong Chen:
    I used QEMU 2.0.0 on Ubuntu 14.04.
    Do you mean your patch can make qemu-img convert qcow2 overlay into a new overlay?

[Lidong Chen (jemmy858585)]

On Mon, Apr 10, 2017 at 1:07 PM, wayen <email address hidden> wrote:
> Hi Lidong Chen:
> I used QEMU 2.0.0 on Ubuntu 14.04.
> Do you mean your patch can make qemu-img convert qcow2 overlay into a new overlay?

yes. but i find it already fixed in 2.0.0.
do you add the -o backing_file= option in the command?

>
> --
> You received this bug notification because you are a member of qemu-
> devel-ml, which is subscribed to QEMU.
> https://bugs.launchpad.net/bugs/1662050
>
> Title:
> qemu-img convert a overlay qcow2 image into a entire image
>
> Status in QEMU:
> Incomplete
>
> Bug description:
> I have a base image file "base.qcow2" and a delta qcow2 image file
> "delta.qcow2" whose backing file is "base.qcow2".
>
> Now I use qemu-img to convert "delta.qcow2" and will get a new image
> file "new.qcow2" which is entire and equivalent to combination of
> "base.qcow2" and "delta.qcow2".
>
> In fact,I don't want to get a complete image.I just want to convert
> delta qcow2 image file "A" to a New delta overlay qcow2 image "B"
> which is equivalent to "A". So the "new.qcow2" is not what i want. I
> have to admit that this is not bug. Could you please take this as a
> new feature and enable qemu-img to convert qcow2 overlay itself?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/qemu/+bug/1662050/+subscriptions
>

[chengkang (chengkang)]

Hi Lidong Chen:

    I forgot to use this option.I think the way you said is effective.I will try it.Thank you very much for your help

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]