Percona XtraBackup moved to https://jira.percona.com/projects/PXB

innobackupex fails on apply-log step

Bug #1423117 reported by dseira on 2015-02-18

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	Percona XtraBackup moved to https://jira.percona.com/projects/PXB	Confirmed	Undecided	Unassigned

Bug Description

The innobackupex script fails to apply-logs with the following error:

innobackupex --apply-log --rebuild-indexes 2015-02-17_15-03-01/

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

IMPORTANT: Please check that the apply-log run completes successfully.
At the end of a successful apply-log run innobackupex
prints "completed OK!".

150218 09:08:02 innobackupex: Starting ibbackup with command: xtrabackup --defaults-file="/tmp/2015-02-17_15-03-01/backup-my.cnf" --defaults-group="mysqld" --prepare --target-dir=/tmp/2015-02-17_15-03-01 --rebuild-indexes

xtrabackup version 2.3.0alpha1 based on MySQL server 5.6.21 Linux (x86_64) (revision id: )
xtrabackup: cd to /tmp/2015-02-17_15-03-01
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(48539303)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
Starting to expand compacted .ibd files.
Expanding ./database2/log.ibd
Expanding ./mysql/slave_master_info.ibd
Expanding ./mysql/innodb_table_stats.ibd
Expanding ./mysql/slave_relay_log_info.ibd
Expanding ./mysql/slave_worker_info.ibd
Expanding ./mysql/innodb_index_stats.ibd
Expanding ./perconatest/heartbeat.ibd
Expanding ./perconatest/example.ibd
Expanding ./database1/XXXXXX.ibd
Expanding ./database1/XXXXXX2.ibd
Expanding ./database1/XXXXXX3.ibd
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup: innodb_data_home_dir = ./
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 1
xtrabackup: innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.3
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: The log sequence numbers 30578769 and 30578769 in ibdata files do not match the log sequence number 48539303 in the ib_logfiles!
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages
InnoDB: from the doublewrite buffer...
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
[01] Checking if there are indexes to rebuild in table mysql/innodb_index_stats (space id: 2)
[01] Checking if there are indexes to rebuild in table mysql/innodb_table_stats (space id: 3)
[01] Checking if there are indexes to rebuild in table mysql/slave_master_info (space id: 4)
[01] Checking if there are indexes to rebuild in table mysql/slave_relay_log_info (space id: 5)
[01] Checking if there are indexes to rebuild in table mysql/slave_worker_info (space id: 6)
[01] Checking if there are indexes to rebuild in table perconatest/example (space id: 163)
[01] Checking if there are indexes to rebuild in table perconatest/heartbeat (space id: 161)
[01] Checking if there are indexes to rebuild in table database1/XXXXX (space id: 321)
[01] Found index Name_XXXXXX
[01] Found index fk_table4_table51_idx
[01] Rebuilding 2 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 322)
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 323)
[01] Found index XXXXXXX
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 324)
[01] Found index Level
[01] Found index Name
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 4 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 325)
[01] Found index XXXXXX
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 327)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 6 index(es).
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_1 (space id: 377)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_2 (space id: 378)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_3 (space id: 379)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_4 (space id: 380)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_5 (space id: 381)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_00000000000007b3_INDEX_6 (space id: 382)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 374)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 375)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 376)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 372)
[01] Checking if there are indexes to rebuild in table database1/FTS_0000000000000181_XXXXXX (space id: 373)
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 351)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 3 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 352)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index fk_XXXXXX_table51_idx
[01] Rebuilding 6 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 353)
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_2
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 5 index(es).
[01] Checking if there are indexes to rebuild in table database1/XXXXXX (space id: 371)
[01] Found index XXXXXX
[01] Found index XXXXXX_cns_1
[01] Found index XXXXXX_cns_2
[01] Found index XXXXXX_cns_3
[01] Found index XXXXXX_cns_4
[01] Found index XXXXXX_cns_8
[01] Found index XXXXXX_cns_9
[01] Found index XXXXXX_cns_5
[01] Found index XXXXXX_cns_6
[01] Found index XXXXXX_cns_7
[01] Found index XXXXXX_cns_10
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_ibfk_7_idx
[01] Found index XXXXXX_idx
[01] Found index XXXXXX_table51_idx
[01] Found index XXXXXX_idx
[01] Rebuilding 29 index(es).
09:08:05 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x2e) [0x912a0e]
xtrabackup(handle_fatal_signal+0x235) [0x819545]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7fce72140cb0]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2616
main::apply_log() called at /usr/bin/innobackupex line 1559
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2616.

The command to generate the backup is:

innobackupex --encrypt=AES256 --encrypt-key-file=/etc/innobackupex.key --compact --compress --user=XXXX --password=XXXXX /tmp

And, after the apply-log step, the decrypt step works perfectly, the command is:

innobackupex --decrypt=AES256 --encrypt-key-file=/etc/innobackupex.key --decompress 2015-02-17_15-03-01

The apply-log command is:

innobackupex --apply-log --rebuild-indexes 2015-02-17_15-03-01

We have tested several percona-xtrabackup version; 2.2.8, 2.2.9 and 2.3-alpha with the same result.

The info about the system is:

percona xtradb cluster: 5.6.21-25.8-938.precise
OS: ubuntu server 12.04.5
percona-xtrabackup: 2.2.8, 2.2.9 and 2.3.0-alpha

I've attached the report for the 2.2.9 version.

Revision history for this message

dseira (davidseira) wrote on 2015-02-18:

apply.log Edit (7.4 KiB, text/plain)

Revision history for this message

dseira (davidseira) wrote on 2015-02-19:

I've continued doing some tests and if we create the backup without the --compact option and if we recover the backup without the --rebuild-indexes after; the backup is recovered perfectly. So it seems a malfunctioning in the rebuild-indexes function.

Revision history for this message

Nilnandan Joshi (nilnandan-joshi) wrote on 2015-02-20:

Download full text (14.2 KiB)

Confirmed with xtrabackup 2.2.9. it seems issue with compact + rebuild-indexes.

nilnandan@desktop:~$ xtrabackup --version
xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
nilnandan@desktop:~$

nilnandan@desktop:~$ sudo innobackupex --user=root --password=root --compact /home/nilnandan/backup/
[sudo] password for nilnandan:

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150220 12:34:49 innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root' (using password: YES).
150220 12:34:49 innobackupex: Connected to MySQL server
150220 12:34:49 innobackupex: Executing a version check against the server...
150220 12:34:53 innobackupex: Done.
150220 12:34:53 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
At the end of a successful backup run innobackupex
prints "completed OK!".

innobackupex: Using server version 5.6.22-72.0

innobackupex: Created backup directory /home/nilnandan/backup/2015-02-20_12-34-53

150220 12:34:53 innobackupex: Starting ibbackup with command: xtrabackup --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/nilnandan/backup/2015-02-20_12-34-53 --innodb_log_file_size="50331648" --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp --extra-lsndir='/tmp' --compact
innobackupex: Waiting for ibbackup (pid=8127) to suspend
innobackupex: Suspend file '/home/nilnandan/backup/2015-02-20_12-34-53/xtrabackup_suspended_2'

Confirmed with xtrabackup 2.2.9.  it seems issue with compact + rebuild-indexes.

nilnandan@desktop:~$ xtrabackup --version
xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
nilnandan@desktop:~$

nilnandan@desktop:~$ sudo innobackupex --user=root --password=root --compact /home/nilnandan/backup/
[sudo] password for nilnandan:

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150220 12:34:49  innobackupex: Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup' as 'root'  (using password: YES).
150220 12:34:49  innobackupex: Connected to MySQL server
150220 12:34:49  innobackupex: Executing a version check against the server...
150220 12:34:53  innobackupex: Done.
150220 12:34:53  innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

innobackupex:  Using server version 5.6.22-72.0

innobackupex: Created backup directory /home/nilnandan/backup/2015-02-20_12-34-53

150220 12:34:53  innobackupex: Starting ibbackup with command: xtrabackup  --defaults-group="mysqld" --backup --suspend-at-end --target-dir=/home/nilnandan/backup/2015-02-20_12-34-53 --innodb_log_file_size="50331648" --innodb_data_file_path="ibdata1:12M:autoextend" --tmpdir=/tmp --extra-lsndir='/tmp' --compact
innobackupex: Waiting for ibbackup (pid=8127) to suspend
innobackupex: Suspend file '/home/nilnandan/backup/2015-02-20_12-34-53/xtrabackup_suspended_2'

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /var/lib/mysql
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 2
xtrabackup:   innodb_log_file_size = 50331648
>> log scanned up to (2834510592)
xtrabackup: Generating a list of tablespaces
[01] Copying ./ibdata1 to /home/nilnandan/backup/2015-02-20_12-34-53/ibdata1
>> log scanned up to (2834510592)
>> log scanned up to (2834510592)
[01]        ...done
>> log scanned up to (2834510592)
[01] Copying ./mysql/slave_relay_log_info.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/mysql/slave_relay_log_info.ibd
[01]        ...done
[01] Copying ./mysql/slave_worker_info.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/mysql/slave_worker_info.ibd
[01]        ...done
[01] Copying ./mysql/innodb_index_stats.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/mysql/innodb_index_stats.ibd
[01]        ...done
[01] Copying ./mysql/slave_master_info.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/mysql/slave_master_info.ibd
[01]        ...done
[01] Copying ./mysql/innodb_table_stats.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/mysql/innodb_table_stats.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_2.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_2.ibd
[01]        ...done
[01] Copying ./nil/test.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/test.ibd
>> log scanned up to (2834510592)
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_4.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_4.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_5.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_5.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_6.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_6.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_DELETED.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_DELETED.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_CONFIG.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_CONFIG.ibd
[01]        ...done
[01] Copying ./nil/test1.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/test1.ibd
>> log scanned up to (2834510592)
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_DELETED_CACHE.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_DELETED_CACHE.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_3.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_3.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_BEING_DELETED.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_BEING_DELETED.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_BEING_DELETED_CACHE.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_BEING_DELETED_CACHE.ibd
[01]        ...done
[01] Copying ./nil/FTS_0000000000000022_0000000000000027_INDEX_1.ibd to /home/nilnandan/backup/2015-02-20_12-34-53/nil/FTS_0000000000000022_0000000000000027_INDEX_1.ibd
[01]        ...done
>> log scanned up to (2834510592)
xtrabackup: Creating suspend file '/home/nilnandan/backup/2015-02-20_12-34-53/xtrabackup_suspended_2' with pid '8128'

150220 12:35:00  innobackupex: Continuing after ibbackup has suspended
150220 12:35:00  innobackupex: Executing LOCK TABLES FOR BACKUP...
150220 12:35:00  innobackupex: Backup tables lock acquired

150220 12:35:00  innobackupex: Starting to backup non-InnoDB tables and files
innobackupex: in subdirectories of '/var/lib/mysql/'
innobackupex: Backing up files '/var/lib/mysql//mysql/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (74 files)
>> log scanned up to (2834510592)
innobackupex: Backing up file '/var/lib/mysql//nil/test.frm'
innobackupex: Backing up file '/var/lib/mysql//nil/db.opt'
innobackupex: Backing up file '/var/lib/mysql//nil/test1.frm'
innobackupex: Backing up files '/var/lib/mysql//performance_schema/*.{frm,isl,MYD,MYI,MAD,MAI,MRG,TRG,TRN,ARM,ARZ,CSM,CSV,opt,par}' (53 files)
150220 12:35:00  innobackupex: Finished backing up non-InnoDB tables and files

150220 12:35:00  innobackupex: Executing LOCK BINLOG FOR BACKUP...
150220 12:35:00  innobackupex: Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS...
150220 12:35:00  innobackupex: Waiting for log copying to finish

xtrabackup: The latest check point (for incremental): '2834510592'
xtrabackup: Stopping log copying thread.
.>> log scanned up to (2834510592)

xtrabackup: Creating suspend file '/home/nilnandan/backup/2015-02-20_12-34-53/xtrabackup_log_copied' with pid '8128'
xtrabackup: Transaction log of lsn (2834510592) to (2834510592) was copied.
150220 12:35:01  innobackupex: Executing UNLOCK BINLOG
150220 12:35:01  innobackupex: Executing UNLOCK TABLES
150220 12:35:01  innobackupex: All tables unlocked

innobackupex: Backup created in directory '/home/nilnandan/backup/2015-02-20_12-34-53'
150220 12:35:01  innobackupex: Connection to database server closed
150220 12:35:01  innobackupex: completed OK!
nilnandan@desktop:~$ 
nilnandan@desktop:~$ 
nilnandan@desktop:~$ 
nilnandan@desktop:~$

-- Apply log

nilnandan@desktop:~$ sudo innobackupex --apply-log --rebuild-indexes /home/nilnandan/backup/2015-02-20_12-34-53/

This software is published under
the GNU GENERAL PUBLIC LICENSE Version 2, June 1991.

Get the latest version of Percona XtraBackup, documentation, and help resources:
http://www.percona.com/xb/p

150220 12:35:49  innobackupex: Starting the apply-log operation

IMPORTANT: Please check that the apply-log run completes successfully.
           At the end of a successful apply-log run innobackupex
           prints "completed OK!".

150220 12:35:49  innobackupex: Starting ibbackup with command: xtrabackup  --defaults-file="/home/nilnandan/backup/2015-02-20_12-34-53/backup-my.cnf"  --defaults-group="mysqld" --prepare --target-dir=/home/nilnandan/backup/2015-02-20_12-34-53 --rebuild-indexes

xtrabackup version 2.2.9 based on MySQL server 5.6.22 Linux (x86_64) (revision id: )
xtrabackup: cd to /home/nilnandan/backup/2015-02-20_12-34-53
xtrabackup: This target seems to be not prepared yet.
xtrabackup: xtrabackup_logfile detected: size=2097152, start_lsn=(2834510592)
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 2097152
Starting to expand compacted .ibd files.
xtrabackup: Generating a list of tablespaces
Expanding ./mysql/slave_worker_info.ibd
Expanding ./mysql/slave_relay_log_info.ibd
Expanding ./mysql/innodb_table_stats.ibd
Expanding ./mysql/slave_master_info.ibd
Expanding ./mysql/innodb_index_stats.ibd
Expanding ./nil/FTS_0000000000000022_CONFIG.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_3.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_1.ibd
Expanding ./nil/FTS_0000000000000022_BEING_DELETED.ibd
Expanding ./nil/test.ibd
Expanding ./nil/FTS_0000000000000022_BEING_DELETED_CACHE.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_5.ibd
Expanding ./nil/test1.ibd
Expanding ./nil/FTS_0000000000000022_DELETED_CACHE.ibd
Expanding ./nil/FTS_0000000000000022_DELETED.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_2.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_4.ibd
Expanding ./nil/FTS_0000000000000022_0000000000000027_INDEX_6.ibd
xtrabackup: using the following InnoDB configuration for recovery:
xtrabackup:   innodb_data_home_dir = ./
xtrabackup:   innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup:   innodb_log_group_home_dir = ./
xtrabackup:   innodb_log_files_in_group = 1
xtrabackup:   innodb_log_file_size = 2097152
xtrabackup: Starting InnoDB instance for recovery.
xtrabackup: Using 104857600 bytes for buffer pool (set by --use-memory parameter)
InnoDB: Using atomics to ref count buffer pool pages
InnoDB: The InnoDB memory heap is disabled
InnoDB: Mutexes and rw_locks use GCC atomic builtins
InnoDB: Memory barrier is not used
InnoDB: Compressed tables use zlib 1.2.8
InnoDB: Using CPU crc32 instructions
InnoDB: Initializing buffer pool, size = 100.0M
InnoDB: Completed initialization of buffer pool
InnoDB: Highest supported file format is Barracuda.
InnoDB: The log sequence numbers 2834510582 and 2834510582 in ibdata files do not match the log sequence number 2834510592 in the ib_logfiles!
InnoDB: Database was not shutdown normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages 
InnoDB: from the doublewrite buffer...
InnoDB: Last MySQL binlog file position 0 330, file name mysql-bin.000002
InnoDB: 128 rollback segment(s) are active.
InnoDB: Waiting for purge to start
[01] Checking if there are indexes to rebuild in table mysql/innodb_index_stats (space id: 2)
[01] Checking if there are indexes to rebuild in table mysql/innodb_table_stats (space id: 1)
[01] Checking if there are indexes to rebuild in table mysql/slave_master_info (space id: 4)
[01] Checking if there are indexes to rebuild in table mysql/slave_relay_log_info (space id: 3)
[01] Checking if there are indexes to rebuild in table mysql/slave_worker_info (space id: 5)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_1 (space id: 21)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_2 (space id: 22)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_3 (space id: 23)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_4 (space id: 24)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_5 (space id: 25)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_0000000000000027_INDEX_6 (space id: 26)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_BEING_DELETED (space id: 29)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_BEING_DELETED_CACHE (space id: 30)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_CONFIG (space id: 31)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_DELETED (space id: 27)
[01] Checking if there are indexes to rebuild in table nil/FTS_0000000000000022_DELETED_CACHE (space id: 28)
[01] Checking if there are indexes to rebuild in table nil/test (space id: 20)
[01]   Found index idx
[01]   Found index FTS_DOC_ID_INDEX
[01]   Rebuilding 2 index(es).
07:05:53 UTC - xtrabackup got signal 11 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x3d) [0x95e67d]
xtrabackup(handle_fatal_signal+0x271) [0x8fe6d1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfc90) [0x7f4460533c90]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2642.
	main::apply_log() called at /usr/bin/innobackupex line 1570
innobackupex: Error: 
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2642.
nilnandan@desktop:~$