innobackupex crashes during sst

[Jesse Sandberg]

I'm running MariaDB-Galera-server 10.0.21 on CentOS 6
and using wsrep_sst_method=xtrabackup-v2

SST works when using:
percona-xtrabackup x86_64 2.2.12-1.el6

SST crashes at donor when using
percona-xtrabackup x86_64 2.3.2-1.el6

xtrabackup's last gasps in innobackup.backup.log :

151030 15:32:58 Finished backing up non-InnoDB tables and files
151030 15:32:58 [00] Streaming xtrabackup_galera_info
151030 15:32:58 [00] ...done
151030 15:32:58 [00] Streaming /var/lib/mysql//blog.000457 to <STDOUT>
151030 15:32:58 [00] ...done
13:32:58 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x8c502e]
innobackupex(handle_fatal_signal+0x273) [0x736dc3]
/lib64/libpthread.so.0() [0x3fe260f790]
/lib64/libc.so.6(gsignal+0x35) [0x3fe2232625]
/lib64/libc.so.6(abort+0x175) [0x3fe2233e05]
/lib64/libc.so.6() [0x3fe2270537]
/lib64/libc.so.6() [0x3fe2275f4e]
/lib64/libc.so.6() [0x3fe2278cad]
innobackupex(write_current_binlog_file(st_mysql*)+0x258) [0x59d678]
innobackupex(backup_start()+0x177) [0x5a3fb7]
innobackupex(xtrabackup_backup_func()+0xc00) [0x58def0]
innobackupex(main+0xaea) [0x591efa]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x3fe221ed5d]
innobackupex() [0x585e79]

[Sergei Glushchenko (sergei.glushchenko)]

Hello,

Could you please post the output of "SHOW MASTER STATUS" and "SHOW VARIABLES".

[Marno (marno.vandermolengmail.com)]

Reporting the same issue on Debian 8 after upgrading to this version of xtrabackup - might be something upstream @ percona? Using 'deb http://repo.percona.com/apt jessie main' in sources.list.

[Jesse Sandberg (jesse-sandberg)]

Another stacktrace included - now with a memory map (if it's of any help)

[Sergei Glushchenko (sergei.glushchenko)]

Thank you Jesse!

From the files you uploaded I made a conclusion that crash is caused by double free of 'datadir' variable in 'write_current_binlog_file' function. This is specific to MariaDB with Galera replication.

The fix will be following:

diff --git i/storage/innobase/xtrabackup/src/backup_mysql.cc w/storage/innobase/xtrabackup/src/backup_mysql.cc
index dbe23cf..6c78fc7 100644
--- i/storage/innobase/xtrabackup/src/backup_mysql.cc
+++ w/storage/innobase/xtrabackup/src/backup_mysql.cc
@@ -1106,7 +1106,7 @@ write_current_binlog_file(MYSQL *connection)
                if (log_bin_dir == NULL) {
                        /* log_bin_basename does not exist in MariaDB,
                        fallback to datadir */
- log_bin_dir = datadir;
+ log_bin_dir = strdup(datadir);
                }

                dirname_part(log_bin_dir, log_bin_dir, &log_bin_dir_length);

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/135

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-445