Feature request: add option to remove *.qp files on decompress

Bug #1446487 reported by Robert Wunderer on 2015-04-21
54
This bug affects 11 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Status tracked in 2.4
2.2
Won't Fix
Undecided
Unassigned
2.3
Fix Released
High
Sergei Glushchenko
2.4
Fix Released
High
Sergei Glushchenko

Bug Description

Since version 2.2.10, as a fix for this bug: https://bugs.launchpad.net/percona-xtrabackup/+bug/1413044, XtraBackup no longer removes *.qp files on decompressing. This can lead to problems when the target filesystem is not big enough for both compressed and decompressed backup files.

The same applies for encrypted files of course.

A solution might be to add an option --remove-compressed and/or --remove-encrypted to enable the user to optionally revert back to the old behaviour in cases where the risk of a corrupted restore is acceptable e.g. because the original backup still exists in another place or the "backup" was the result of a hotcopy from a live host.

Thank you for the feature request.

Changed in percona-xtrabackup:
status: New → Confirmed
MikeyH (mikeyh) wrote :

This would help a lot as I was trying to figure out why restored backups were so much bigger and we ran out of disk space.

Brendon (brendon-h) wrote :

This actually caused a recent restore to fail for me because the drive filled. Luckily it was only for staging and not production.

Alexander Rubin (alikrubin75) wrote :

As a workaround (BE CAREFULL with this, as it will remove qp files, but should work fine):
cd your_backup_dir
find . -type f -name "*.qp" -exec rm -v {} \;

This is good for removing the files after decompressing has finished, but in our case we need to remove each single file immediately after it has been decompressed because there is simply not enough free space on our fusion io card for both compressed and uncompressed version of the data files. (Same problem as Brendon's, probably)

For now we'll stick with version 2.2.9 of xtrabackup, the last version to still have exactly this behaviour.

Xtrabackup will be removing *.qp and *.xbcrypt files once we implement this https://blueprints.launchpad.net/percona-xtrabackup/+spec/checksum-unencrypted-chunk

Nathan Anderson (4athan) wrote :

We found when you combine the following snippet:

https://github.com/percona/percona-xtrabackup/blob/2.2/storage/innobase/xtrabackup/innobackupex.pl#L1578-L1597

with this snippet:

https://github.com/percona/percona-xtrabackup/commit/efd4a9142f77298a705be8390baee9b0586e166c

You end up with the backups being decrypted+decompressed, then decrypted again, then probably decompressed again, since the source files are never deleted.

We noticed this because we were using omghuge amounts of space when decompressing/decrypting our backups. We ended up writing our own decompress/decrypt routines so we could restore in-situ instead of using the sub-optimal innobackupex routines.

I'm +1 on figuring out a way to avoid the duplicate decompression/decryption cycles.

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-424

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers