Feature request: add option to remove *.qp files on decompress

[Robert Wunderer]

Since version 2.2.10, as a fix for this bug: https://bugs.launchpad.net/percona-xtrabackup/+bug/1413044, XtraBackup no longer removes *.qp files on decompressing. This can lead to problems when the target filesystem is not big enough for both compressed and decompressed backup files.

The same applies for encrypted files of course.

A solution might be to add an option --remove-compressed and/or --remove-encrypted to enable the user to optionally revert back to the old behaviour in cases where the risk of a corrupted restore is acceptable e.g. because the original backup still exists in another place or the "backup" was the result of a hotcopy from a live host.

[Valerii Kravchuk (valerii-kravchuk)]

Thank you for the feature request.

[MikeyH (mikeyh)]

This would help a lot as I was trying to figure out why restored backups were so much bigger and we ran out of disk space.

[Bren (uitmater)]

This actually caused a recent restore to fail for me because the drive filled. Luckily it was only for staging and not production.

[Alexander Rubin (alikrubin75)]

As a workaround (BE CAREFULL with this, as it will remove qp files, but should work fine):
cd your_backup_dir
find . -type f -name "*.qp" -exec rm -v {} \;

[Robert Wunderer (robert-wunderer-y)]

This is good for removing the files after decompressing has finished, but in our case we need to remove each single file immediately after it has been decompressed because there is simply not enough free space on our fusion io card for both compressed and uncompressed version of the data files. (Same problem as Brendon's, probably)

For now we'll stick with version 2.2.9 of xtrabackup, the last version to still have exactly this behaviour.

[Sergei Glushchenko (sergei.glushchenko)]

Xtrabackup will be removing *.qp and *.xbcrypt files once we implement this https://blueprints.launchpad.net/percona-xtrabackup/+spec/checksum-unencrypted-chunk

[Nathan Anderson (4athan)]

We found when you combine the following snippet:

https://github.com/percona/percona-xtrabackup/blob/2.2/storage/innobase/xtrabackup/innobackupex.pl#L1578-L1597

with this snippet:

https://github.com/percona/percona-xtrabackup/commit/efd4a9142f77298a705be8390baee9b0586e166c

You end up with the backups being decrypted+decompressed, then decrypted again, then probably decompressed again, since the source files are never deleted.

We noticed this because we were using omghuge amounts of space when decompressing/decrypting our backups. We ended up writing our own decompress/decrypt routines so we could restore in-situ instead of using the sub-optimal innobackupex routines.

I'm +1 on figuring out a way to avoid the duplicate decompression/decryption cycles.

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/300
https://github.com/percona/percona-xtrabackup/pull/301

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-424