xtrabackup: error: log block numbers mismatch

Reported by Oleg M on 2012-01-17
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Percona XtraBackup
Critical
Alexey Kopytov
2.0
Critical
Alexey Kopytov

Bug Description

Running innobackupex I see the following in its output:

 xtrabackup: error: log block numbers mismatch:
 xtrabackup: error: expected log block no. 91292509, but got no. 90768229 from the log file.
 xtrabackup: Error: xtrabackup_copy_logfile() failed.

 innobackupex continues, but at the end I see this:

 innobackupex: xtrabackup_55 failed! (exit code 1) The backup may not be complete.

 Studying Bug #805593 I understood that one possible reason for this error message could be transaction log being wrapped around while xtrabackup works.

 Will much appretiate if you answer the following:
 1) Is it really transaction log wrapping issue?
 Is there a way to find out?
 2) If the above is true, how the backup logic should be build?
 Restart backup?
 Shall this go to FAQ/User Guide?
 3) Is it possible to delay log wrapping till xtrabackup ends?

 I'm using
 Ubuntu Natty 11.04(2.6.38-8-server #42-Ubuntu SMP Mon Apr 11 03:49:04 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux)
 Percona Server 5.5.17-rel22.1
 XTRABACKUP_VERSION=1.6.3

 Please let me know if you need any further details to investgate the issue.

 Thank you and best regards

Related branches

Alexey Kopytov (akopytov) wrote :

Thank you for the bug report. This problem is a regression introduced in the 2.0 branch by the fix for bug #805593. I will commit a fix for it shortly.

krisdba (krisdba) wrote :

It seems this bug stil repeatable in xtrabackup-2.0.
Iam using percona-xtrabackup-2.0.0-417.rhel5 and trying to remotely backup the largest database running on Percona 5.5.16.

xtrabackup: error: Log block checksum mismatch (block no 933830996 at lsn 1028145714688):
expected 997077168, calculated checksum 438936379
xtrabackup: Error: xtrabackup_copy_logfile() failed.

xtrabackup: Error: log_copying_thread failed.

Any help would be appreciated.

Thank you

Alexey Kopytov (akopytov) wrote :

That seems to be a different problem. A checksum (rather than number) mismatch means that the checksum calculated by XtraBackup for a log block does not match whatever is stored in the block itself. This could either be a corrupted log due to a hardware or FS failure, or an XtraBackup bug. We don't have any similar reports, so I'm inclined to think it's a hardware problem. Is there anything in the kernel log?

krisdba (krisdba) wrote :

Thanks Alexey,
There is nothing suspicious reported in system and kernel logs.
No warning/errors in MySQL logs on object corruption.
Unfortunately, this errors is repeatable and backup is failing and we will not be able to recover the data.
Is there a way to troubleshoot what caused it. Appreciate your help.

Alexey Kopytov (akopytov) wrote :

I think I found the reason. When XtraBackup reads a log block that has not been completely written by the server yet, it will fail with an error like that. Which is a regression introduced in 2.0. XtraBackup 1.6 will print a warning, but will retry reading the same block later.

I reported this problem as bug #1015416.

krisdba (krisdba) wrote :

Thanks for the details. appreciate your time.

Quan Tong Anh (anhquankitty) wrote :

- percona-xtrabackup-2.0.1-446.rhel5
- datadir ~ 80GB

xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. x, but got no. y from the log file.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

xtrabackup: Error: log_copying_thread failed.
120719 17:28:44 innobackupex: All tables unlocked
120719 17:28:44 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/data/var/lib/mysql'
innobackupex: MySQL binlog position: filename 'mysql-bin.000994', position 316435839
innobackupex: You must use -i (--ignore-zeros) option for extraction of the tar stream.
120719 17:28:44 innobackupex: xtrabackup_55 failed! (exit code 1) The backup may not be complete.

Bernd Brodda (bbrodda) wrote :

Same here:

- InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy and Percona Inc 2009-2012. All Rights Reserved.
- xtrabackup_55 version 2.0.1 for Percona Server 5.5.16 Linux (x86_64) (revision id: 446)
- MySQL-Server version: 5.5.24-55-log Percona Server (GPL), Release rel26.0, Revision 256
- act as a slave
- Datadir ~81GB

.
.
.
xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 537104441, but got no. 537124913 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

.
.
.
xtrabackup: The latest check point (for incremental): '1374609880770'
xtrabackup: Stopping log copying thread.

xtrabackup: Error: log_copying_thread failed.
120719 12:55:21 innobackupex: All tables unlocked
120719 12:55:21 innobackupex: Connection to database server closed

innobackupex: Backup created in directory '/mnt/2012-07-19_11-52-59'
innobackupex: MySQL binlog position: filename 'shop4-slavedb-bin.000004', position 107
innobackupex: MySQL slave binlog position: master host 'shop4-masterdb-int', filename 'shop4-masterdb.002048', position 420950703
120719 12:55:21 innobackupex: xtrabackup_55 failed! (exit code 1) The backup may not be complete.

Quan Tong Anh (anhquankitty) wrote :

I've been trying the `innobackupex` script in the source code folder (pulling with `bzr branch lp:percona-xtrabackup`) and got the same error:

xtrabackup: error: log block numbers mismatch:
xtrabackup: error: expected log block no. 744382184, but got no. 744402656 from the log file.
xtrabackup: error: it looks like InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.
xtrabackup: Error: xtrabackup_copy_logfile() failed.

Alexey Kopytov (akopytov) wrote :

Bernd, Quan,

The problems you have reported appear to be different from the original one reported in this bug.

In both cases it's probably what the error message says: the InnoDB log has wrapped around before xtrabackup could process all records due to either log copying being too slow, or log files being too small.

Please try increasing the log file size.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Related questions