Getting error while taking percona incremental backup

[santhosh]

Getting the below error
2016-02-01 10:21:53 7fde8e398700 InnoDB: Assertion failure in thread 140593845602048 in file ut0mem.cc line 105
InnoDB: Failing assertion: ret || !assert_on_error
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:21:54 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

[santhosh (santhosh-stalin)]

pls check the log attached .
Error:

InnoDB: Failing assertion: ret || !assert_on_error
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
10:19:44 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...

[santhosh (santhosh-stalin)]

Other details:
My server version is CentOS Linux release 7.0.1406 (Core)
Database version is Server version: 5.6.23 MySQL Community Server (GPL)
ibdata1 is 76M in size

[Sergei Glushchenko (sergei.glushchenko)]

How much of available memory do you have? This error is malloc error.

[Sergei Glushchenko (sergei.glushchenko)]

Also what options do you start the backup with? Using --parallel requires more memory for buffers.

[santhosh (santhosh-stalin)]

Options:
 innobackupex --incremental <path to save> --user=backup_user --password=<password> --incremental-basedir=<physical backup path>

Note: while taking physical dump am not facing this issue ,only during incremental backups am facing this error.

I have scripted using shell
For Full backup:
innobackupex --user=$USER_NAME --password=$PASS_WORD $BCK_PATH >&${BACKUP_LOG}

For incr backup:
innobackupex --incremental $BCK_PATH --user=$USER_NAME --password=$PASS_WORD --incremental-basedir=$PREVIOUS_BCK >&${BACKUP_LOG}

Ram:
 free -mt
             total used free shared buffers cached
Mem: 1678 1495 183 56 14 72
-/+ buffers/cache: 1408 270
Swap: 0 0 0
Total: 1678 1495 183

[Sergei Glushchenko (sergei.glushchenko)]

Can you by chance produce the core dump or resolved stacktrace? You will need to install package with debug symbols.

The command looks OK. How many tablespaces (i.e. *.ibd files) are located in the datadir?

[santhosh (santhosh-stalin)]

the data size is less.PFB the contents of data dir:

total 2.8G
drwx------ 2 mysql mysql 4.0K Feb 16 2015 test
drwx------ 2 mysql mysql 4.0K Feb 16 2015 performance_schema
drwx------ 2 mysql mysql 4.0K Feb 16 2015 mysql
-rw-rw---- 1 mysql mysql 56 Feb 16 2015 auto.cnf
-rw-rw---- 1 mysql mysql 1.4G Nov 4 12:19 ib_logfile1
-rw-rw---- 1 mysql mysql 6 Nov 4 12:19 atdb01s.pid
drwx------ 2 mysql mysql 4.0K Nov 18 10:00 DBA_Utility
drwx------ 2 mysql mysql 52K Feb 2 11:31 stageat
-rw-rw---- 1 mysql mysql 76M Feb 2 12:36 ibdata1
-rw-rw---- 1 mysql mysql 1.4G Feb 2 12:36 ib_logfile0

[santhosh (santhosh-stalin)]

Team,

Pls update on this,

Thanks.

[santhosh (santhosh-stalin)]

160208 08:03:45 innobackupex: Starting the backup operation

IMPORTANT: Please check that the backup run completes successfully.
           At the end of a successful backup run innobackupex
           prints "completed OK!".

Can't locate Digest/MD5.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .) at - line 693.
BEGIN failed--compilation aborted at - line 693.
160208 08:03:46 Connecting to MySQL server host: localhost, user: backup_user, password: set, port: 3306, socket: /AZVOL/mysql/tmp/mysql.sock
Using server version 5.6.23
innobackupex version 2.3.3 based on MySQL server 5.6.24 Linux (x86_64) (revision id: 525ca7d)
incremental backup from 6808886287 is enabled.
xtrabackup: uses posix_fadvise().
xtrabackup: cd to /AZVOL/mysql/data
xtrabackup: open files limit requested 0, set to 1024
xtrabackup: using the following InnoDB configuration:
xtrabackup: innodb_data_home_dir = /AZVOL/mysql/data
xtrabackup: innodb_data_file_path = ibdata1:12M:autoextend
xtrabackup: innodb_log_group_home_dir = ./
xtrabackup: innodb_log_files_in_group = 2
xtrabackup: innodb_log_file_size = 1415577600
160208 08:03:47 >> log scanned up to (6808886377)
160208 08:03:49 >> log scanned up to (6808886377)
xtrabackup: Generating a list of tablespaces
160208 08:03:51 >> log scanned up to (6808886377)
160208 08:03:53 >> log scanned up to (6808886377)
160208 08:03:54 >> log scanned up to (6808886377)
160208 08:03:56 >> log scanned up to (6808886377)
160208 08:03:58 >> log scanned up to (6808886377)
160208 08:04:00 >> log scanned up to (6808886377)
160208 08:04:03 >> log scanned up to (6808886377)
160208 08:04:05 >> log scanned up to (6808886377)
160208 08:04:06 >> log scanned up to (6808886377)
160208 08:04:09 >> log scanned up to (6808886377)
160208 08:04:10 >> log scanned up to (6808886377)
160208 08:04:12 >> log scanned up to (6808886377)
160208 08:04:13 >> log scanned up to (6808886377)
160208 08:04:15 >> log scanned up to (6808886377)
160208 08:04:17 >> log scanned up to (6808886377)
160208 08:04:18 >> log scanned up to (6808886377)
160208 08:04:19 >> log scanned up to (6808886377)
160208 08:04:22 >> log scanned up to (6808886377)
160208 08:04:23 >> log scanned up to (6808886377)
160208 08:04:25 >> log scanned up to (6808886377)
160208 08:04:26 >> log scanned up to (6808886377)
160208 08:04:28 >> log scanned up to (6808886377)
160208 08:04:29 >> log scanned up to (6808886377)
160208 08:04:30 >> log scanned up to (6808886377)
160208 08:04:32 >> log scanned up to (6808886377)
160208 08:04:35 >> log scanned up to (6808886377)
160208 08:04:37 >> log scanned up to (6808886377)
160208 08:04:38 >> log scanned up to (6808886377)
160208 08:04:40 >> log scanned up to (6808886377)
xtrabackup: using the full scan for incremental backup
160208 08:04:42 >> log scanned up to (6808886377)
2016-02-08 08:04:42 7ffaf2df8700 InnoDB: Assertion failure in thread 140715793286912 in file ut0mem.cc line 105
InnoDB: Failing assertion: ret || !assert_on_error
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com....

[Sergei Glushchenko (sergei.glushchenko)]

Have you managed to get the core dump or stack traces?
You need to install debug symbols for xtrabackup and enable core dumps. Once you have core dump you need to run GDB and point it to the core dump. Then run "bt" and "t apply bt full".

Instead of enabling core dumps you can run gdb --args <your backup command> and then, when it crashed just execute the commands "bt" and "t apply bt full".

I am afraid I am of little help without repeatable test case or resolved stack trace.

[Sergei Glushchenko (sergei.glushchenko)]

also I don't see how it affects security

[santhosh (santhosh-stalin)]

I am able to take core dump , my script for taking full backup works fine , only in incremental backup i am facing this.
Let me try with 4G RAM and check if its related to malloc.

[santhosh (santhosh-stalin)]

I tried gdb:

 gdb --args innobackupex --incremental /AZVOL/phy_backup/inc --user=backup_user --password=XXXXXXXX --incremental-basedir=/AZVOL/phy_backup/2016-02-08_07-52-26/
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/bin/xtrabackup...Reading symbols from /usr/bin/xtrabackup...(no debugging symbols found)...done.
(no debugging symbols found)...done.
Missing separate debuginfos, use: debuginfo-install percona-xtrabackup-2.3.3-1.el7.x86_64
(gdb)
(gdb)
(gdb) bt
No stack.
(gdb) t apply bt full
(gdb)
(gdb) quit

[santhosh (santhosh-stalin)]

This issue is due to malloc only when I tried with 4G of RAM there was no issue,
Thanks percona support team.
Pls close this incident

[Launchpad Janitor (janitor)]

[Expired for Percona XtraBackup because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-1367