[Feature Request] support xz for compressing volumes

[Rob Fortune]

xz offers a superior compression ratio at the cost of higher CPU load during creation of backup, but still decompresses (when you need it most) at a rapid pace. I don't know if it should be the default but adding it as an option could save yottabytes of bandwidth :)

[Rob Fortune (usedonlytosignup)]

I have been in discussion with the authors of xz and because of the way it works there is no ideal settings for compression. Simply throwing -1 or -9 at it like bzip2/gzip is wrong.

They attest the best compression value would be -6e because it only requires 9MB of ram to decompress and you don't lose a lot of compression.

[Mechanical snail (replicator-snail)]

The choice of compression algorithm should be configurable. I might use lzop or gzip for backups locally or over a fast network, and xz over a slow internet uplink.

In any case, on slow connections, it's not that important to choose the optimal settings, as practically anything will be better than gzip but still fast enough to saturate the internet connection. (xz -9 -e achieves 0.5 MB/s on my laptop, which is sufficient for a residential internet connection.)

[Ryan Kitty (gothickitty93)]

Is this still an issue that is being worked on? It should be marked as "wishlist" if it could still be done, or marked as "won't fix" if it will not be implemented

[Lapse of Reason (lapseofreason0)]

I've started to implement this for unencrypted volumes, but I've run into a few issues that maybe someone could help me with.

The first problem is that lzma.LZMAFile from pylzma doesn't support using a fileobj instead of a real file, does someone know a way to work around that?

Then maybe one of the duplicity developers can point me to where decompression is done in the code. I had no problems finding compression in gpg.py and I first thought decompression was done using tarfile.py, but I'm not so sure any more.

[Lapse of Reason (lapseofreason0)]

Here is a patch that implements compressing of duplicity archives using lzma (i.e. xz). It requires python-lzma to be installed.

Please be aware that decompressing is not yet implemented, so there is no way to restore such archives yet. Furthermore I have not tested this, so be aware that you might lose data if you use it.

I would appreciate if someone could help test this. Moreover I still couldn't figure out where in the code decompression is done, so I'd appreciate hints in regard to that so I can implement the rest.

[Lapse of Reason (lapseofreason0)]

I've just uploaded my code. There are still two remaining issues:

First of all there is an error that prevents restoration of xz compressed archives that I haven't been able to track down. If anyone knows how to fix this, please let me know.

Then there is also a warning during backup which I'm not sure how to get rid of, but it doesn't seem to affect the backup.

Again, I'd appreciate any help finalizing and testing this.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "duplicity-lzma-write.patch" of this bug report has been identified as being a patch. The ubuntu-reviewers team has been subscribed to the bug report so that they can review the patch. In the event that this is in fact not a patch you can resolve this situation by removing the tag 'patch' from the bug report and editing the attachment so that it is not flagged as a patch. Additionally, if you are member of the ubuntu-sponsors please also unsubscribe the team from this bug report.

[This is an automated message performed by a Launchpad user owned by Brian Murray. Please contact him regarding any issues with the action taken in this bug report.]

[Laszlo Ersek (lacos-caesar)]

(Well I'm quite late to this bug, but I think it would be preferable to add an "external filter" option, to be forked in-between the rdiffdir delta and gpg. This would enable all kinds of (preferably multi-threaded) compressors, without introducing more library dependencies.)