qcow2 image increasing disk size above the virtual limit

[Todor Andreev]

Using qemu/kvm, qcow2 images, ext4 file systems on both guest and host
 Host and Guest: Ubuntu server 12.04 64bit
To create an image I did this:

qemu-img create -f qcow2 -o preallocation=metadata ubuntu-pdc-vda.img 10737418240 (not sure about the exact bytes, but around this)
ls -l ubuntu-pdc-vda.img
fallocate -l theSizeInBytesFromAbove ubuntu-pdc-vda.img

The problem is that the image is growing progressively and has obviously no limit, although I gave it one. The root filesystem's image is the same case:

qemu-img info ubuntu-pdc-vda.img
 image: ubuntu-pdc-vda.img
 file format: qcow2
 virtual size: 10G (10737418240 bytes)
 disk size: 14G
 cluster_size: 65536

and for confirmation:
 du -sh ubuntu-pdc-vda.img
 15G ubuntu-pdc-vda.img

I made a test and saw that when I delete something from the guest, the real size of the image is not decreasing (I read it is normal). OK, but when I write something again, it doesn't use the freed space, but instead grows the image. So for example:
 1. The initial physical size of the image is 1GB.
 2. I copy 1GB of data in the guest. It's physical size becomes 2GB.
 3. I delete this data (1GB). The physical size of the image remains 2GB.
 4. I copy another 1GB of data to the guest.
 5. The physical size of the image becomes 3GB.
 6. And so on with no limit. It doesn't care if the virtual size is less.

Is this normal - the real/physical size of the image to be larger than the virtual limit???

[Serge Hallyn (serge-hallyn)]

Thanks for filing this bug, Todor. I'll try figure out whether this is still the case in the upstream git HEAD.

[Serge Hallyn (serge-hallyn)]

I started playing with this by just doing:

  qemu-img create -f qcow2 x.img 2G
  (boot a vm from a cdrom/iso into rescue mode with x.img as a drive, and there do):
  dd if=/dev/zero of=/mnt/zero1 bs=1M count=1000
and then
  cp /mnt/zero1 /mnt/zero2
  rm /mnt/zero2
  cp /mnt/zero1 /mnt/zero3
  (etc)

Here the volume doesn't exceed it's allocation (which was 2G).

I created snapshots but still did not exceed 2G.

When I started adding more real data (booted from an installed server virtual disk, but I don't believe that made a difference) as well as creating snapshots, I worked up to a 3.2G real disk.

In the disk you showed in the bug description, had you created any snapshots?

[Todor Andreev (toshko3)]

Yes, I have created one snapshot and did fallocate in the beginning. The other image, which I have problems with, also has snapshots.

[Serge Hallyn (serge-hallyn)]

First going back to the original bug, in that instance you kept around many snapshots. In that case there is no way to avoid having many snapshots of, say, a 2G disk, taking much more space than 2G.

The thing that concerned me in this bug was that disk space was never reclaimed.

I don't believe that this is the case when no snapshots are used. If you create a new qcow2 image with 2G size, then that image will not exceed 2G on disk.

Once you introduce snapshots, this appears to complicate the bookkeeping such that automatic resizing of the disk image is not done. The data *is* reference counted however, This means that you can create a new, trimmed qcow2 image based on the original by doing

qemu-img convert -f qcow2 -O qcow2 original.qcow2 new.qcow2

As I don't believe the automatic freeing of disk space was ever implemented, I am going to mark this bug Triaged/Low, to mark it as a desirable feature. I'll also mark it as affecting the upstream project.

[Todor Andreev (toshko3)]

I did some testing with a WindowsXP guest, that I have and could test on.
It seems that this behavior is not present at the beginning. But at the moment we create a snapshot it is starting to write on top of the current size. So, it is like this:
 1. Image is:

Code:
qemu-img info WindowsXP.img
image: WindowsXP.img
file format: qcow2
virtual size: 15G (15728640000 bytes)
disk size: 13G
cluster_size: 65536

I write some files and it doesn't become more than that.

 2. I delete some files, then write again and it doesn't changes size.
 3. I create a snapshot:

Code:
qemu-img info WindowsXP.img
image: WindowsXP.img
file format: qcow2
virtual size: 15G (15728640000 bytes)
disk size: 13G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 test 0 2012-07-17 09:52:25 00:00:00.000

4. I write something with a size of 587MB and it becomes larger (with 587MB)

Code:
du -sm WindowsXP.img
14102 WindowsXP.img

5. I delete it and then write it again. It becomes more larger... (again with another 587MB)

Code:
du -sm WindowsXP.img
14703 WindowsXP.img

6. I delete it and then write it again. It doesn't change this time.

7. I write a copy of it. It becomes larger (with 587MB)

Code:
du -sm WindowsXP.img
15309 WindowsXP.img

8. I delete it and then write it again. It doesn't change this time.

9. I write a copy of it. It becomes larger again (with 587MB)

Code:
du -sm WindowsXP.img
16010 WindowsXP.img

10. I write another copy of it and it stays the same.

Code:
du -sm WindowsXP.img
16010 WindowsXP.img

11. I write another copy of it and becomes larger again.

Code:
du -sm WindowsXP.img
16913 WindowsXP.img

Code:
qemu-img info WindowsXP.img
image: WindowsXP.img
file format: qcow2
virtual size: 15G (15728640000 bytes)
disk size: 17G
cluster_size: 65536
Snapshot list:
ID TAG VM SIZE DATE VM CLOCK
1 test 0 2012-07-17 09:52:25 00:00:00.000

12. Create a snapshot.
13. I write a copy of it. It becomes larger again (with 587MB)

Code:
du -sm WindowsXP.img
17572 WindowsXP.img

14. I delete both of the snapshots.
15. I delete the file (in guest with the size of 587MB) and write it again. No change in size.
16. I delete the file again and write it again. No change in size.
17. Create a snapshot.
18. I delete the file again and write it again. No change in size.
19. Delete the file.
20. Create a snapshot.
21. Write the file again. No change in size.

 Well from all this, I can conclude that most probably:
 1. The problem occurs only when there is an internal snapshot present.
 2. The problem is not "by design" because the behavior is not consistent (for example, 13. and 21. should be with the same result, but they arent't)..

At the end of the day, after these procedures (4 creations of snapshots, 2 deletions and some writing and deleting of internal guest files) the result is this:

Code:
qemu-img info WindowsXP.img
image: WindowsXP.img
file format: qcow2
virtual size: 15G (15...

[Todor Andreev (toshko3)]

May be here will be more convenient for reading:
http://www.linuxquestions.org/questions/linux-virtualization-90/disk-physical-size-more-than-virtual-size-qcow2-image-4175416848/#post4730524

[Todor Andreev (toshko3)]

@Serge, thank you for answering.
Well, I think this is not entirely true. I think that from my above post, 13. and 20. should be with the same results, if it were true. The truth is that when a snapshot is present, sometimes it uses the available space, sometimes it doesn't. (!???)
I think there is something wrong here and everyone should be able to confirm it, because I did it on 2 different hosts, with 2 different guest images.
Apart from this, I now understand that when I use internal snapshots, the size will be larger (it is logical).

Also (apart from my different 13. and 20. results, pointing that there is something wrong), I think that this is a HIGH priority, because the problem is with an image of almost 2TB. So, what people should buy another 2TB, so that they could convert the image somewhere? I don't think this is reasonable.

[Todor Andreev (toshko3)]

I'm sorry, I meant 21. not 20.

[Serge Hallyn (serge-hallyn)]

@Todor,

Thanks, you might be right. It sounds like it's not a missing feature but a bug. I'll re-raise the priority.

[Andy Menzel (menzean)]

Any solution right now? I have a similar problem like Todor Andreev;
Our daily backup of some virtual machines (qcow2) looks like that:

1. shutdown the VM
2. create a snapshot via: "qemu-img snapshot -c nameofsnapshot..."
3. boot the VM
4. backup the snapshot to another virtual disk via: "qemu-img convert -f qcow2 -O qcow2 -s nameofsnapshot..."
5. DELETE the snapshot from VM via: qemu-img snapshot -d nameofsnapshot...

But the problem is, that our original VM-size growing steadily (although few changes were made) ?!

[Serge Hallyn (serge-hallyn)]

I don't know of any qcow2-based workaround.

Is anyone actively working on fixing the qcow2 code? In particular, the fact that after removing snapshots, un-used blocks are not reclaimed and disk size is never reduced?

One possible workaround (the one I would use) would be to use lvm-based snapshotting instead.

[Stefan Hajnoczi (stefanha)]

On Tue, Dec 18, 2012 at 10:18:20AM -0000, Andy Menzel wrote:
> Any solution right now? I have a similar problem like Todor Andreev;
> Our daily backup of some virtual machines (qcow2) looks like that:
>
> 1. shutdown the VM
> 2. create a snapshot via: "qemu-img snapshot -c nameofsnapshot..."
> 3. boot the VM
> 4. backup the snapshot to another virtual disk via: "qemu-img convert -f qcow2 -O qcow2 -s nameofsnapshot..."
> 5. DELETE the snapshot from VM via: qemu-img snapshot -d nameofsnapshot...

It's not safe to modify the qcow2 file while the guest is running. This
means Step 5 is not really safe and could result in an inconsistent
image.

This may also be causing the problem: the QEMU process has a variable
with the next free cluster index. Since Step 5 runs as a separate
process it does not update the QEMU process' next free cluster index
variable. QEMU doesn't know that there are now free clusters within the
image file because you updated the file behind QEMU's back - the result
is that it grows the file.

Please try deleting the last backup snapshot between Step 1 and Step 2.
This way you'll free the space while QEMU isn't accessing the image
file. When you boot up the image file again QEMU should reuse the freed
clusters.

Stefan

[Eric Blake (eblake)]

On 01/02/2013 08:50 AM, Stefan Hajnoczi wrote:
> On Tue, Dec 18, 2012 at 10:18:20AM -0000, Andy Menzel wrote:
>> Any solution right now? I have a similar problem like Todor Andreev;
>> Our daily backup of some virtual machines (qcow2) looks like that:
>>
>> 1. shutdown the VM
>> 2. create a snapshot via: "qemu-img snapshot -c nameofsnapshot..."
>> 3. boot the VM
>> 4. backup the snapshot to another virtual disk via: "qemu-img convert -f qcow2 -O qcow2 -s nameofsnapshot..."
>> 5. DELETE the snapshot from VM via: qemu-img snapshot -d nameofsnapshot...
>
> It's not safe to modify the qcow2 file while the guest is running. This
> means Step 5 is not really safe and could result in an inconsistent
> image.
>
> This may also be causing the problem: the QEMU process has a variable
> with the next free cluster index. Since Step 5 runs as a separate
> process it does not update the QEMU process' next free cluster index
> variable. QEMU doesn't know that there are now free clusters within the
> image file because you updated the file behind QEMU's back - the result
> is that it grows the file.
>
> Please try deleting the last backup snapshot between Step 1 and Step 2.
> This way you'll free the space while QEMU isn't accessing the image
> file. When you boot up the image file again QEMU should reuse the freed
> clusters.

You might also want to try modifying step 5 to use the HMP delvm monitor
command from within the running qemu rather than going behind qemu's
back with a qemu-img invocation. That's how libvirt deletes internal
snapshots from a running qemu.

Also, there are patches currently under review that are talking about
creating a QMP counterpart to the delvm monitor command.

--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[Andy Menzel (menzean)]

Thanks for your advices. I have no more problems with VM-size since deleting snapshot in shutdown-mode. I reduced the overlarge qcow2-images by converting in qcow2 again (that detects unused sectors and omits this).

[Mario (q-mario)]

Is anyone even looking at this? been years and the problem still persists!

[Michael Tokarev (mjt+launchpad-tls)]

Looking at what? At the lack of problems as comment #14 says?

[Serge Hallyn (serge-hallyn)]

Changing priority given workarounds.

[Mario (q-mario)]

@michael, so you do that once, after some time the machine keeps growing, and growing and growing... and you have to redo that every so often... I have a machine that should be taking up 30 gb yet is taking 600+ GB with 4 snapshots... but yeah... I'll just plug in another 1tb hard drive so that i can free up the space only for it to happen again in a near future... Seems a great workaround!

[Serge Hallyn (serge-hallyn)]

For the record, the workaround is deleting old snapshots in shutdown mode
as per comment #14.

Upstream has moved toward external snapshots as the way forward, so while
I don't argue that this is a bug, it seems unlikely to receive a fix from
upstream.

[Max Reitz (xanclic)]

@Mario, in theory an image "that should be taking up 30 GB" with four snapshots should be taking up at most about 150 GB, of course. Now the question is what you mean by "should be taking up 30 GB" and by "is taking 600+ GB".

For the latter, did you query the file length (ls -l) or the actual size (qemu-img info, "disk size")?

For the former, if you have a virtual disk size of 1 TB and the guest reports 30 GB are used, that doesn't mean that qemu knows that only 30 GB are used. If you delete a file in the guest, it will report less space being used; however, qemu doesn't know about that unless the guest bothers to discard the now unused sectors. If it doesn't (and I don't see a reason why a guest should discard sectors on an HDD), the guest will just remove the file metadata but the data will stay there (and may be overwritten later by the guest when creating new files etc.). qemu has no idea that that data is now unused, therefore it must treat those sectors as being in use.

If your image indeed has a virtual disk size of 30 GB, has four snapshots, is clean (qemu-img check) and does take up 600+ GB of actual disk space, that should indeed not be happening (unless there's some case I forgot to consider).

[Mario (q-mario)]

@serge, what version would I need to upgrade to be able to use the external snapshots? that sounds like it would solve my problems

[Serge Hallyn (serge-hallyn)]

@Mario,

the external snapshots have apparently been around a long time. The
ability to create external snapshots from running vms is newer, but
it appears to exist evn in qemu-kvm 1.0. So all versions in Debian
and Ubuntu should support them.

http://wiki.qemu.org/Features/Snapshots#Snapshot_command_flow

[Thomas Huth (th-huth)]

Looking through old bug tickets... is there anything left to do here? Or should we rather close this ticket nowadays?

[Launchpad Janitor (janitor)]

[Expired for qemu-kvm (Ubuntu) because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for QEMU because there has been no activity for 60 days.]