Add on-line write compression support to qcow2

[Timothy Miller]

This is a wishlist item. Launchpad really need a way for the submitter to indicate this.

It would be really cool if qemu were to support disk compression on-line for writes.

I know this wouldn't be really easy. Although most OS's use blocks, you can really only count on being able to compress 512-byte sectors, which doesn't give much room for a good compression ratio. Moreover, the index indicating where in the image file each sector is located would be complex to manage, since the compressed blocks would be variable sized, and you'd be wanting to do some kind of best-fit allocation of space in the image file. (If you were to make the image file compressed block size granularity, say, 64 bytes, you could probably do this best fit O(1).) If you were to buffer enough writes, you could group arbitrary sequences of written sectors into blocks to compress (which with writeback could be sent to a helper thread on another CPU, so the throughput would be good).

[Paul Maunders (sq2d3bipy0t2o-paul)]

+1 vote for this feature.

[Thomas Huth (th-huth)]

As far as I know, QEMU v5.1 now has support for compression filters, e.g. by creating a qcow2 image with:

 qemu-img create -f qcow2 -o compression_type=zlib image.qcow2 1G

... so I think we can finally mark this ticket here as done.

[Eric Blake (eblake)]

On 11/19/20 3:39 AM, Thomas Huth wrote:
> As far as I know, QEMU v5.1 now has support for compression filters,
> e.g. by creating a qcow2 image with:
>
> qemu-img create -f qcow2 -o compression_type=zlib image.qcow2 1G
>
> ... so I think we can finally mark this ticket here as done.

That says what compression type to use when writing the entire disk in
one pass, but not online write compression. I think we may be a bit
premature in calling this 'fix released', although I'm not certain we
will ever try to add the feature requested.

>
> ** Changed in: qemu
> Status: Confirmed => Fix Released
>

--
Eric Blake, Principal Software Engineer
Red Hat, Inc. [phone]
Virtualization: qemu.org | libvirt.org

[Thomas Huth (th-huth)]

Ok, sorry, seems like I mis-understood that new compression_type feature. If the requested feature will likely never be implemented, should we move this to WontFix instead?

[Max Reitz (xanclic)]

The compression filter can be used e.g. with -drive driver=compress,file.driver=qcow2,file.file.filename=foo.qcow2. However, it shouldn’t be used lightly, as it will only do the right thing in very specific circumstances, namely every cluster that’s written to must not be allocated already. So writing to the same cluster twice will not work. (Which is why I was hesitant to merge this filter, but in the end I was contend with the fact that it’s at least difficult enough to use that unsuspecting users hopefully won’t accidentally enable it.)

(It should be noted that this is not a limitation of the compression filter, though, but of qcow2’s implementation (VMDK isn’t any better). So technically qemu has the feature now, but qcow2 is still missing it.)