innobackupex crashes on --apply-log

[Ilia Chipitsine]

that bug is reproducible. contact me at <email address hidden> , I can provide you with access to that server

[root@localhost innob]# cat /etc/redhat-release
CentOS Linux release 7.1.1503 (Core)

[root@localhost innob]# rpm -qa | grep xtrabackup
percona-xtrabackup-2.2.11-1.el7.x86_64

[root@localhost innob]# innobackupex --apply-log 2015-07-27_23-03-31/

InnoDB Backup Utility v1.5.1-xtrabackup; Copyright 2003, 2009 Innobase Oy
and Percona LLC and/or its affiliates 2009-2013. All Rights Reserved.

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150801 00:01:38 innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150801 00:01:38 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/home/ilia/itpanda/innob/2015-07-27_23-03-31/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/home/ilia/itpanda/innob/2015-07-27_23-03-31

xtrabackup version 2.2.11 based on MySQL server 5.6.24 Linux (x86_64) (revision id: )
xtrabackup: cd to /home/ilia/itpanda/innob/2015-07-27_23-03-31
xtrabackup: This target seems to be not prepared yet.
2015-08-01 00:01:38 7f65aee31740 InnoDB: Operating system error number 2 in a file operation.
InnoDB: The error means the system cannot find the path specified.
xtrabackup: Warning: cannot open ./xtrabackup_logfile. will try to find.
  xtrabackup: 'ib_logfile0' seems to be 'xtrabackup_logfile'. will retry.
xtrabackup: xtrabackup_logfile detected: size=537575424, start_lsn=(256227499037)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 537575424
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:10M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 537575424
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.7
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: Log scan progressed past the checkpoint lsn 256227499037
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: Doing recovery: scanned up to log sequence number 256232741888 (1%)
InnoDB: Doing recovery: scanned up to log sequence number 256237984768 (2%)
InnoDB: Doing recovery: scanned up to log sequence number 256243227648 (3%)
InnoDB: Doing recovery: scanned up to log sequence number 256248470528 (4%)
InnoDB: Doing recovery: scanned up to log sequence number 256253713408 (5%)
InnoDB: Doing recovery: scanned up to log sequence number 256258956288 (6%)
InnoDB: Doing recovery: scanned up to log sequence number 256264199168 (7%)
InnoDB: Doing recovery: scanned up to log sequence number 256269442048 (8%)
InnoDB: Doing recovery: scanned up to log sequence number 256274684928 (9%)
InnoDB: Doing recovery: scanned up to log sequence number 256279927808 (10%)
InnoDB: Doing recovery: scanned up to log sequence number 256285170688 (12%)
InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percent: 0 2015-08-01 00:04:20 7f65a577e700 InnoDB: Assertion failure in thread 140074544523008 in file log0recv.cc line 1236
InnoDB: Failing assertion: !page || (ibool)!!page_is_comp(page) == dict_table_is_comp(index->table)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
19:04:20 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x35) [0xa2d717]
xtrabackup(handle_fatal_signal+0x2ab) [0x7a6613]
/lib64/libpthread.so.0(+0xf130) [0x7f65aea2e130]
/lib64/libc.so.6(gsignal+0x37) [0x7f65ad19b5d7]
/lib64/libc.so.6(abort+0x148) [0x7f65ad19ccc8]
xtrabackup() [0x70aa6d]
xtrabackup(recv_recover_page_func(unsigned long, buf_block_t*)+0x3cd) [0x70bc1f]
xtrabackup(buf_page_io_complete(buf_page_t*)+0x44a) [0x6b9c74]
xtrabackup(fil_aio_wait(unsigned long)+0x146) [0x69d126]
xtrabackup(io_handler_thread+0x25) [0x73c04d]
/lib64/libpthread.so.0(+0x7df5) [0x7f65aea26df5]
/lib64/libc.so.6(clone+0x6d) [0x7f65ad25c1ad]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2649.
        main::apply_log() called at /usr/bin/innobackupex line 1578
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2649.
[root@localhost innob]#

[Muhammad Irfan (muhammad-irfan)]

Can you please share exact commands used for backup and to prepare backup. Also, please provide the source version of MySQL where you took the backup.
mysql> SHOW VARIABLES LIKE '%version%';

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1339