xtrabackup xtrabackup: error: last checkpoint LSN (XXX) is larger than last copied LSN (XXX).
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Percona XtraBackup moved to https://jira.percona.com/projects/PXB |
Expired
|
Undecided
|
Unassigned |
Bug Description
I use xtrabackup my percona server, it's a rather heavy server with over thounds of qps, the backup complete successfully, however the recovery failed. Or the backup process does not complete at all
here is the backup command and recovery command, ( both of them are on the same machine) Version:2.3.3 Linux-Generic
OS: Amazon Linux 4.1.13-
Hareware: AWS EC2 r3.2xlarge
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 8
On-line CPU(s) list: 0-7
Thread(s) per core: 2
Core(s) per socket: 4
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Stepping: 4
CPU MHz: 2494.084
BogoMIPS: 4988.16
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-7
Total memory: 61GB
Storage : 160GB SSD of Instance Storage
BACKUP:
innobackupex --user=root --host=127.0.0.1 --password=PASSWD --port=3308 --defaults-
innobackupex --user=root --host=127.0.0.1 --password=PASSWD --port=3308 --defaults-
here is the backup command failure output :
151231 13:46:27 Finished backing up non-InnoDB tables and files
151231 13:46:27 [00] Writing xtrabackup_
151231 13:46:27 [00] ...done
151231 13:46:27 Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
xtrabackup: The latest check point (for incremental): '256243027607'
xtrabackup: Stopping log copying thread.
.151231 13:46:27 >> log scanned up to (256228180480)
151231 13:46:27 Backup created in directory '/data/
MySQL binlog position: filename 'mysql-bin.000166', position '497999141'
151231 13:46:27 [00] Writing backup-my.cnf
151231 13:46:27 [00] ...done
151231 13:46:27 [00] Writing xtrabackup_info
151231 13:46:27 [00] ...done
xtrabackup: Transaction log of lsn (256228180557) to (256228180480) was copied.
xtrabackup: error: last checkpoint LSN (256243027607) is larger than last copied LSN (256228180480).
when the backup completed successfully, I tried to run recovery command :
innobackupex --apply-log /data/dbbackup/
151231 13:55:14 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
innobackupex version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
xtrabackup: cd to /data/dbbackup/
xtrabackup: This target seems to be not prepared yet.
2015-12-31 13:55:14 7f49b5fbd740 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_
xtrabackup: xtrabackup_logfile detected: size=235814912, start_lsn=
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 255720678541
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 255725921280 (2%)
InnoDB: Doing recovery: scanned up to log sequence number 255731164160 (5%)
InnoDB: Doing recovery: scanned up to log sequence number 255736407040 (7%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 255741649920 (10%)
InnoDB: Doing recovery: scanned up to log sequence number 255746892800 (12%)
InnoDB: Doing recovery: scanned up to log sequence number 255752135680 (15%)
InnoDB: Doing recovery: scanned up to log sequence number 255757378560 (17%)
.
.
.
.
.2 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 255799321600 (37%)
InnoDB: Doing recovery: scanned up to log sequence number 255804564480 (40%)
InnoDB: Doing recovery: scanned up to log sequence number 255809807360 (42%)
InnoDB: Doing recovery: scanned up to log sequence number 255815050240 (45%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 255820293120 (47%)
InnoDB: Doing recovery: scanned up to log sequence number 255825536000 (50%)
InnoDB: Doing recovery: scanned up to log sequence number 255830778880 (52%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99
InnoDB: Apply batch completed
InnoDB: Doing recovery: scanned up to log sequence number 255836021760 (55%)
InnoDB: Doing recovery: scanned up to log sequence number 255841264640 (57%)
InnoDB: Doing recovery: scanned up to log sequence number 255846507520 (60%)
InnoDB: Doing recovery: scanned up to log sequence number 255851553280 (62%)
InnoDB: We scanned the log up to 255851553280. A checkpoint was at 255720678541 and the maximum LSN on a database page was 256051447089. It is possible that the database is now corrupt!
2015-12-31 13:55:21 7f49b5fbd740 InnoDB: Error: page 4 log sequence number 255865085692
InnoDB: is in the future! Current system log sequence number 255851553183.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://
InnoDB: for more information.
2015-12-31 13:55:21 7f49b5fbd740 InnoDB: Error: page 5 log sequence number 255922379541
InnoDB: is in the future! Current system log sequence number 255851553183.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://
InnoDB: for more information.
2015-12-31 13:55:21 7f49b5fbd740 InnoDB: Error: page 6 log sequence number 255922311331
.
.
.
.
.
.
...skipping...
InnoDB: for more information.
2015-12-31 13:55:22 7f49ac4da700 InnoDB: Error: page 780 log sequence number 255907590246
InnoDB: is in the future! Current system log sequence number 255851553183.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://
InnoDB: for more information.
2015-12-31 13:55:22 7f49ac4da700 InnoDB: Error: page 790 log sequence number 255937400281
InnoDB: is in the future! Current system log sequence number 255851553183.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://
InnoDB: for more information.
InnoDB: Apply batch completed
2015-12-31 13:55:22 7f49b5fbd740 InnoDB: Error: page 5 log sequence number 255922379541
InnoDB: is in the future! Current system log sequence number 255851553183.
InnoDB: Your database may be corrupt or you may have copied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB: http://
InnoDB: for more information.
InnoDB: Cleaning up trx with id 1870338525
2015-12-31 13:55:22 7f49b5fbd740 InnoDB: Assertion failure in thread 139954562520896 in file trx0trx.cc line 292
InnoDB: Failing assertion: trx->update_undo == NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://
InnoDB: about forcing recovery.
13:55:22 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(
innobackupex(
/lib64/
/lib64/
/lib64/
innobackupex(
innobackupex(
innobackupex(
innobackupex() [0x5798ae]
innobackupex(
/lib64/
innobackupex(
Please report a bug at https:/
description: | updated |
summary: |
- Back and Recovery does not work + xtrabackup xtrabackup: error: last checkpoint LSN (XXX) is larger than + last copied LSN (XXX). |
Is your backup fails always or it's failed first time with error ? Did you re-tried to backup ? Please, provide server my.cnf file and output of backup directory. 2015-12- 31_13-45- 10/
$ ls -lah /data/dbbackup/