xtrabackup have error log block numbers mismatch

[Eugene Chetverikov]

Good day!

When using the tool (with a similar load) sometimes get the error:

xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 217388165, but got no. 250942589 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: xtrabackup_copy_logfile () failed.
innobackupex: got a fatal error with the following stacktrace: at / usr / local / bin / innobackupex line 2694.
         main :: wait_for_ibbackup_file_create ('/ mnt / backup / autocheck / xtrabackup_suspended_2') called at / usr / local / bin / innobackupex line 2714
         main :: wait_for_ibbackup_suspend ('/ mnt / backup / autocheck / xtrabackup_suspended_2') called at / usr / local / bin / innobackupex line 1968
         main :: backup () called at / usr / local / bin / innobackupex line 1592
innobackupex: Error: The xtrabackup child process has died at / usr / local / bin / innobackupex line 2694.

I tried to run a backup with a similar load of about 15 times
In some cases, the backup is successful (approx. 50%)
Transmission media - 1 Gbit/s
Database size about 900Gb
innodb_log_file_size = 8192M

InnoDB Backup Utility v1.5.1-xtrabackup
xtrabackup version 2.2.7
(binaries from http://www.percona.com/downloads/XtraBackup/XtraBackup-2.2.7/binary)

Percona server ver 5.6.21.70.0 (gentoo portages)

[Alexey Kopytov (akopytov)]

Eugene,

Some back-of-the-envelope calculations show that with the given database size, innodb_log_file_size and transfer bandwidth the redo log size can be overrun with the average redo log write rate of just 2.25 MB/sec. Which doesn't look much, as in there are good chances your average load during the backup is higher in 50% cases.

Do you have an estimate on the average redo generation rate on that server? You can find tips on how to figure it out in this post: http://www.percona.com/blog/2012/10/08/measuring-the-amount-of-writes-in-innodb-redo-logs/

[Eugene Chetverikov (leidruid)]

I used smaller log file size (slightly larger than required, as described in article http://www.percona.com/blog/2008/11/21/how-to-calculate-a-good-innodb- log-file-size /), but the percentage of failed attempts was greater (~ 80%)

[Jaime Sicam (jssicam)]

Hi Eugene,
  Does this problem still occur if you perform the backup from Xtrabackup 2.2.5?

[Jaime Sicam (jssicam)]

It seems we're getting the same errors except that "xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small." is not printed out:

Eg #1
xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 637407195, but got no. 638455767 from the log file.
xtrabackup: Error: xtrabackup_copy_logfile() failed.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2695.

Eg #2
xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 647432068, but got no. 648480640 from the log file.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

This problem seems to be occurring on Xtrabackup 2.2.6 and 2.2.7

[Eugene Chetverikov (leidruid)]

Hi, Jaime!
I have that problem with any version

[zcy (554185217-f)]

Hi
I have that problem with 2.0.8 ,when I backup vary large data (1.6T) with xbstream to remote host.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1308