Comment 17 for bug 1177206

Revision history for this message
akrus (akrus-flygroup) wrote :

I'm not sure if my problem is the same as described here, but I experience similar things with latest Percona Xtrabackup 2.2 series.

Here's how backup was created:
innobackupex --compact --compress --compress-threads=6 --parallel=6 --slave-info --stream=xbstream /var/tmp/mysql

Further things that were done:
1) xbstream -x
2) innobackupex --decompress
3a) innobackupex --apply-log
3b) innobackupex --apply-log --rebuild-indexes --use-memory=20G

(first time I was only doing apply-log and thought the problem is not rebuild-indexes, but it seems it doesn't change anything)

So the output is:
InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

141128 08:23:39 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

141128 08:23:39 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/home/mysql/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/mysql --use-memory=
20G --rebuild-indexes

xtrabackup version 2.2.6 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: cd to /home/mysql
xtrabackup: This target seems to be not prepared yet.
2014-11-28 08:23:39 7f123becb780 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
  xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=36615536640, start_lsn=(163422478633685)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 36615536640
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:100M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 36615536640
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 21474836480 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.8
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 20.0G
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 163422478633685
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 163422483876352 (0%)
...
InnoDB: Doing recovery: scanned up to log sequence number 163432235633152 (29%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 2014-11-28 08:29:40 7f0d0145d700 InnoDB: Assertion failure in thread 139693832656640 in file log0recv.cc line 1268
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
06:29:40 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x93454e]
xtrabackup(handle_fatal_signal+0x262) [0x8f51c2]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340) [0x7f123baa8340]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39) [0x7f123a20cbb9]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f123a20ffc8]
xtrabackup() [0x5c20c8]
xtrabackup(recv_recover_page_func(unsigned long, buf_block_t*)+0x509) [0x5c4239]
xtrabackup(buf_page_io_complete(buf_page_t*)+0x410) [0x6bd620]
xtrabackup(fil_aio_wait(unsigned long)+0x138) [0x64db98]
xtrabackup(io_handler_thread+0x28) [0x654718]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x8182) [0x7f123baa0182]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f123a2d0fbd]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2632.
        main::apply_log() called at /usr/bin/innobackupex line 1561
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2632.

As I've tried dumping the whole database two times and it's quite large (~700Gb compressed without secondary indexes) - this takes some time. Both times the same error was appearing. What's wrong with it?