I'm not sure if my problem is the same as described here, but I experience similar things with latest Percona Xtrabackup 2.2 series.
Here's how backup was created:
innobackupex --compact --compress --compress-threads=6 --parallel=6 --slave-info --stream=xbstream /var/tmp/mysql
Further things that were done:
1) xbstream -x
2) innobackupex --decompress
3a) innobackupex --apply-log
3b) innobackupex --apply-log --rebuild-indexes --use-memory=20G
(first time I was only doing apply-log and thought the problem is not rebuild-indexes, but it seems it doesn't change anything)
So the output is:
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources: http://www.percona.com/xb/p
141128 08:23:39 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: cd to /home/mysql
xtrabackup: This target seems to be not prepared yet.
2014-11-28 08:23:39 7f123becb780 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=36615536640, start_lsn=(163422478633685)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 36615536640
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 36615536640
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 21474836480 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.8
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 20.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 163422478633685
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 163422483876352 (0%)
...
InnoDB: Doing recovery: scanned up to log sequence number 163432235633152 (29%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2014-11-28 08:29:40 7f0d0145d700 InnoDB: Assertion failure in thread 139693832656640 in file log0recv.cc line 1268
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:29:40 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x93454e]
xtrabackup(handle_fatal_signal+0x262) [0x8f51c2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f123baa8340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f123a20cbb9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f123a20ffc8]
xtrabackup() [0x5c20c8]
xtrabackup(recv_recover_page_func(unsigned long, buf_block_t*)+0x509) [0x5c4239]
xtrabackup(buf_page_io_complete(buf_page_t*)+0x410) [0x6bd620]
xtrabackup(fil_aio_wait(unsigned long)+0x138) [0x64db98]
xtrabackup(io_handler_thread+0x28) [0x654718]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f123baa0182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f123a2d0fbd]
Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2632. main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2632.
As I've tried dumping the whole database two times and it's quite large (~700Gb compressed without secondary indexes) - this takes some time. Both times the same error was appearing. What's wrong with it?
I'm not sure if my problem is the same as described here, but I experience similar things with latest Percona Xtrabackup 2.2 series.
Here's how backup was created: threads= 6 --parallel=6 --slave-info --stream=xbstream /var/tmp/mysql
innobackupex --compact --compress --compress-
Further things that were done:
1) xbstream -x
2) innobackupex --decompress
3a) innobackupex --apply-log
3b) innobackupex --apply-log --rebuild-indexes --use-memory=20G
(first time I was only doing apply-log and thought the problem is not rebuild-indexes, but it seems it doesn't change anything)
So the output is:
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.
This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.
Get the latest version of Percona XtraBackup, documentation, and help resources: www.percona. com/xb/ p
http://
141128 08:23:39 innobackupex: Starting the apply-log operation
IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".
141128 08:23:39 innobackupex: Starting ibbackup with command: xtrabackup --defaults- file="/ home/mysql/ backup- my.cnf" --defaults- group=" mysqld" --prepare --target- dir=/home/ mysql --use-memory=
20G --rebuild-indexes
xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: ) logfile. will try to find. logfile' . will retry. (16342247863368 5) data_home_ dir = ./ data_file_ path = ibdata1: 100M:autoextend log_group_ home_dir = ./ log_files_ in_group = 1 log_file_ size = 36615536640 data_home_ dir = ./ data_file_ path = ibdata1: 100M:autoextend log_group_ home_dir = ./ log_files_ in_group = 1 log_file_ size = 36615536640 !!page_ is_comp( page) == dict_table_ is_comp( index-> table) bugs.mysql. com. dev.mysql. com/doc/ refman/ 5.6/en/ forcing- innodb- recovery. html
xtrabackup: cd to /home/mysql
xtrabackup: This target seems to be not prepared yet.
2014-11-28 08:23:39 7f123becb780 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_
xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_
xtrabackup: xtrabackup_logfile detected: size=36615536640, start_lsn=
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: innodb_
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 21474836480 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.8
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 20.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 163422478633685
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 163422483876352 (0%)
...
InnoDB: Doing recovery: scanned up to log sequence number 163432235633152 (29%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2014-11-28 08:29:40 7f0d0145d700 InnoDB: Assertion failure in thread 139693832656640 in file log0recv.cc line 1268
InnoDB: Failing assertion: !page || (ibool)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://
InnoDB: about forcing recovery.
06:29:40 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Thread pointer: 0x0 my_print_ stacktrace+ 0x2e) [0x93454e] handle_ fatal_signal+ 0x262) [0x8f51c2] 64-linux- gnu/libpthread. so.0(+0x10340) [0x7f123baa8340] 64-linux- gnu/libc. so.6(gsignal+ 0x39) [0x7f123a20cbb9] 64-linux- gnu/libc. so.6(abort+ 0x148) [0x7f123a20ffc8] recv_recover_ page_func( unsigned long, buf_block_ t*)+0x509) [0x5c4239] buf_page_ io_complete( buf_page_ t*)+0x410) [0x6bd620] fil_aio_ wait(unsigned long)+0x138) [0x64db98] io_handler_ thread+ 0x28) [0x654718] 64-linux- gnu/libpthread. so.0(+0x8182) [0x7f123baa0182] 64-linux- gnu/libc. so.6(clone+ 0x6d) [0x7f123a2d0fbd]
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(
xtrabackup(
/lib/x86_
/lib/x86_
/lib/x86_
xtrabackup() [0x5c20c8]
xtrabackup(
xtrabackup(
xtrabackup(
xtrabackup(
/lib/x86_
/lib/x86_
Please report a bug at https:/ /bugs.launchpad .net/percona- xtrabackup innobackupex line 2632.
main:: apply_log( ) called at /usr/bin/ innobackupex line 1561 innobackupex line 2632.
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/
As I've tried dumping the whole database two times and it's quite large (~700Gb compressed without secondary indexes) - this takes some time. Both times the same error was appearing. What's wrong with it?