Long-lasting backup of active databases fails with "Combined size of log files must be < 512 GB"

Bug #1425269 reported by Eike Frost
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup moved to https://jira.percona.com/projects/PXB
Fix Released
High
Sergei Glushchenko
2.2
Fix Released
High
Sergei Glushchenko
2.3
Fix Released
High
Sergei Glushchenko

Bug Description

When backing up a somewhat large, disk-backed MariaDB installation with active writes going in (~10TiB in compressed size, backup taking several days to finish, and somewhat busy writes during that time), unforeseen trouble arises due to the limitation of log files to 512GiB; XtraBackup does not warn of this problem when creating the backup (claiming everything went OK), but the backup is unusable due to --apply-log being impossible to do. This is not related to bug #1264432 (while the prepare process does increase this logfile size by 1/8 on every attempt, even the base backup had a logfile of > 512GiB).

~~~~
$ innobackupex --apply-log --export .

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150224 12:10:16 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150224 12:10:16 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/dbdata/tmpbackup/backup-my.cnf" --defaults-group="mysqld" --prepare --target-
dir=/dbdata/tmpbackup --export

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: auto-enabling --innodb-file-per-table due to the --export option
xtrabackup: cd to /dbdata/tmpbackup
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=747591827456, start_lsn=(27657099615443)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 747591827456
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 747591827456
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2642.
        main::apply_log() called at /usr/bin/innobackupex line 1570
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2642.
~~~~

XtraBackup should, at the very least, inform of this problem when creating the backup and not suggest that everything is OK (only to discover down the line that the backup is unusable for recovery).

I am not sure whether it is possible to somehow split log-files and apply them one after another to circumvent this limitation in InnoDB.

Fortunately for me I can partition the backups since this installation uses files_per_table and several databases and thus I can reduce the timeframe where writes need to captured in the log, reducing it to below 455G (accounting for the 1/8 margin required for redo-log-space added at prepare-time). Still this could be a very nasty surprise and is at least annoying when it happens unexpectedly.

Kenny Gryp (gryp)
tags: added: i51823
tags: added: i52198
Revision history for this message
Jericho Rivera (jericho-rivera) wrote :

Customer is affected by this bug

> ls -al xtrabackup_logfile
-rw-r-----. 1 root root 634942029824 Mar 24 00:48 xtrabackup_logfile

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150323 18:40:45 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150323 18:40:45 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/db/p/urpt/v3/table/URPTP1/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/db/p/urpt/v3/table/URPTP1 --use-memory=30GB

xtrabackup version 2.2.8 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: cd to /db/p/urpt/v3/table/URPTP1
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=1287209385984, start_lsn=(129633916825627)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:200M:autoextend:max:50G
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 1287209385984
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:200M:autoextend:max:50G
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 1287209385984
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 32212254720 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 30.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Combined size of log files must be < 512 GB
xtrabackup: innodb_init(): Error occured.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2633
        main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2633.

Changed in percona-xtrabackup:
status: New → Confirmed
Changed in percona-xtrabackup:
assignee: nobody → Sergei Glushchenko (sergei.glushchenko)
importance: Undecided → High
Changed in percona-xtrabackup:
status: Confirmed → Triaged
tags: added: i53523
Revision history for this message
Sergei Glushchenko (sergei.glushchenko) wrote :
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-421

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.