Long-lasting backup of active databases fails with "Combined size of log files must be < 512 GB"
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB |
Fix Released
|
High
|
Sergei Glushchenko | ||
2.2 |
Fix Released
|
High
|
Sergei Glushchenko | ||
2.3 |
Fix Released
|
High
|
Sergei Glushchenko |
Bug Description
When backing up a somewhat large, disk-backed MariaDB installation with active writes going in (~10TiB in compressed size, backup taking several days to finish, and somewhat busy writes during that time), unforeseen trouble arises due to the limitation of log files to 512GiB; XtraBackup does not warn of this problem when creating the backup (claiming everything went OK), but the backup is unusable due to --apply-log being impossible to do. This is not related to bug #1264432 (while the prepare process does increase this logfile size by 1/8 on every attempt, even the base backup had a logfile of > 512GiB).
~~~~
$ innobackupex --apply-log --export .
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources:
http://
150224 12:10:16 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
150224 12:10:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-
dir=/dbdata/
xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: auto-enabling --innodb-
xtrabackup: cd to /dbdata/tmpbackup
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=747591827456, start_lsn=
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/
~~~~
XtraBackup should, at the very least, inform of this problem when creating the backup and not suggest that everything is OK (only to discover down the line that the backup is unusable for recovery).
I am not sure whether it is possible to somehow split log-files and apply them one after another to circumvent this limitation in InnoDB.
Fortunately for me I can partition the backups since this installation uses files_per_table and several databases and thus I can reduce the timeframe where writes need to captured in the log, reducing it to below 455G (accounting for the 1/8 margin required for redo-log-space added at prepare-time). Still this could be a very nasty surprise and is at least annoying when it happens unexpectedly.
tags: | added: i51823 |
tags: | added: i52198 |
Changed in percona-xtrabackup: | |
status: | New → Confirmed |
Changed in percona-xtrabackup: | |
assignee: | nobody → Sergei Glushchenko (sergei.glushchenko) |
importance: | Undecided → High |
Changed in percona-xtrabackup: | |
status: | Confirmed → Triaged |
tags: | added: i53523 |
Customer is affected by this bug
> ls -al xtrabackup_logfile
-rw-r-----. 1 root root 634942029824 Mar 24 00:48 xtrabackup_logfile
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources: www.percona. com/xb/ p
http://
150323 18:40:45 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
150323 18:40:45 innobackupex: Starting ibbackup with command: xtrabackup --defaults- file="/ db/p/urpt/ v3/table/ URPTP1/ backup- my.cnf" --defaults- group=" mysqld" --prepare --target- dir=/db/ p/urpt/ v3/table/ URPTP1 --use-memory=30GB
xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: ) v3/table/ URPTP1 (12963391682562 7) data_home_ dir = ./ data_file_ path = ibdata1: 200M:autoextend :max:50G log_group_ home_dir = ./ log_files_ in_group = 1 log_file_ size = 1287209385984 data_home_ dir = ./ data_file_ path = ibdata1: 200M:autoextend :max:50G log_group_ home_dir = ./ log_files_ in_group = 1 log_file_ size = 1287209385984 innobackupex line 2633
main:: apply_log( ) called at /usr/bin/ innobackupex line 1561 innobackupex line 2633.
xtrabackup: cd to /db/p/urpt/
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=1287209385984, start_lsn=
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 32212254720 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 30.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/