Long-lasting backup of active databases fails with "Combined size of log files must be < 512 GB"

[Eike Frost]

When backing up a somewhat large, disk-backed MariaDB installation with active writes going in (~10TiB in compressed size, backup taking several days to finish, and somewhat busy writes during that time), unforeseen trouble arises due to the limitation of log files to 512GiB; XtraBackup does not warn of this problem when creating the backup (claiming everything went OK), but the backup is unusable due to --apply-log being impossible to do. This is not related to bug #1264432 (while the prepare process does increase this logfile size by 1/8 on every attempt, even the base backup had a logfile of > 512GiB).

~~~~
$ innobackupex --apply-log --export .

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150224 12:10:16 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150224 12:10:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/dbdata/tmpbackup/backup-my.cnf" --defaults-group="mysqld" --prepare --target-
dir=/dbdata/tmpbackup --export

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: auto-enabling --innodb-file-per-table due to the --export option
xtrabackup: cd to /dbdata/tmpbackup
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=747591827456, start_lsn=(27657099615443)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 747591827456
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 747591827456
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2642.
        main::apply_log() called at /usr/bin/innobackupex line 1570
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2642.
~~~~

XtraBackup should, at the very least, inform of this problem when creating the backup and not suggest that everything is OK (only to discover down the line that the backup is unusable for recovery).

I am not sure whether it is possible to somehow split log-files and apply them one after another to circumvent this limitation in InnoDB.

Fortunately for me I can partition the backups since this installation uses files_per_table and several databases and thus I can reduce the timeframe where writes need to captured in the log, reducing it to below 455G (accounting for the 1/8 margin required for redo-log-space added at prepare-time). Still this could be a very nasty surprise and is at least annoying when it happens unexpectedly.

[Jericho Rivera (jericho-rivera)]

Customer is affected by this bug

> ls -al xtrabackup_logfile
-rw-r-----. 1 root root 634942029824 Mar 24 00:48 xtrabackup_logfile

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150323 18:40:45 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150323 18:40:45 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/db/p/urpt/v3/table/URPTP1/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/db/p/urpt/v3/table/URPTP1 --use-memory=30GB

xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: cd to /db/p/urpt/v3/table/URPTP1
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=1287209385984, start_lsn=(129633916825627)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:200M:autoextend:max:50G
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 1287209385984
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:200M:autoextend:max:50G
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 1287209385984
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 32212254720 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 30.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2633
        main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2633.

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/46
https://github.com/percona/percona-xtrabackup/pull/47

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-421