XtraBackup 2.2.x crash on prepare when backing up MySQL/Percona Server 5.5.x

[Kolbe]

Using XtraBackup 2.2.5.

When preparing a backup that was taken using the --databases option to limit the backup to a subset of InnoDB tables, XtraBackup crashes when preparing, This does not occur when the --databases option is omitted from the initial backup.

2014-12-02 18:39:02 7ffe8f0f5720 InnoDB: Assertion failure in thread 140731298567968 in file pars0pars.cc line 865
InnoDB: Failing assertion: sym_node->table != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
01:39:02 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x2186eb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x35) [0x9f2a9c]
xtrabackup(handle_fatal_signal+0x2bb) [0x7f5cd3]
/lib64/libpthread.so.0() [0x386180f710]
/lib64/libc.so.6(gsignal+0x35) [0x3861032635]
/lib64/libc.so.6(abort+0x175) [0x3861033e15]
xtrabackup() [0x76b100]
xtrabackup(pars_update_statement(upd_node_t*, sym_node_t*, void*)+0x30) [0x76ba28]
xtrabackup(yyparse()+0xcb1) [0xa5c6b7]
xtrabackup(pars_sql(pars_info_t*, char const*)+0xaf) [0x76d1bd]
xtrabackup(que_eval_sql(pars_info_t*, char const*, unsigned long, trx_t*)+0x85) [0x78dfea]
xtrabackup(row_drop_table_for_mysql(char const*, trx_t*, bool, bool)+0xa98) [0x71fb84]
xtrabackup(row_mysql_drop_temp_tables()+0x24c) [0x72067b]
xtrabackup(recv_recovery_rollback_active()+0x2c) [0x752ff8]
xtrabackup(innobase_start_or_create_for_mysql()+0x17aa) [0x72855c]
xtrabackup() [0x606fae]
xtrabackup() [0x60f6ba]
xtrabackup(main+0xc33) [0x610e5f]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x386101ed5d]
xtrabackup() [0x603fb9]

[Alexey Kopytov (akopytov)]

Most likely a duplicate of bug #1383062 and has probably the same root cause as server bug #1257266.

George, can you confirm this and update this report when you make progress with bug #1257266?

[Jervin R (revin)]

Still happening on 2.2.10, source backup is 5.5.34 PS, 2.1.9 PXB:

2015-04-18 17:43:21 7ffa5f593720 InnoDB: Assertion failure in thread 140713318233888 in file pars0pars.cc line 865
InnoDB: Failing assertion: sym_node->table != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
22:43:21 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x1d9deb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x35) [0x9f5331]
xtrabackup(handle_fatal_signal+0x2bb) [0x7f801b]
/lib64/libpthread.so.0() [0x376c00f710]
/lib64/libc.so.6(gsignal+0x35) [0x376bc32925]
/lib64/libc.so.6(abort+0x175) [0x376bc34105]
xtrabackup() [0x76bfb0]
xtrabackup(pars_update_statement(upd_node_t*, sym_node_t*, void*)+0x30) [0x76c8d8]
xtrabackup(yyparse()+0xcb1) [0xa5ef27]
xtrabackup(pars_sql(pars_info_t*, char const*)+0xaf) [0x76e06d]
xtrabackup(que_eval_sql(pars_info_t*, char const*, unsigned long, trx_t*)+0x85) [0x78eeb2]
xtrabackup(row_drop_table_for_mysql(char const*, trx_t*, bool, bool)+0xa98) [0x720a0c]
xtrabackup(row_mysql_drop_temp_tables()+0x24c) [0x721503]
xtrabackup(recv_recovery_rollback_active()+0x2c) [0x753ebe]
xtrabackup(innobase_start_or_create_for_mysql()+0x17aa) [0x7293c4]
xtrabackup() [0x607a00]
xtrabackup() [0x610204]
xtrabackup(main+0x8b8) [0x611674]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x376bc1ed1d]
xtrabackup() [0x604369]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
innobackupex: got a fatal error with the following stacktrace: at /usr/bin/innobackupex line 2641
        main::apply_log() called at /usr/bin/innobackupex line 1570
innobackupex: Error:
innobackupex: ibbackup failed at /usr/bin/innobackupex line 2641.

[Fungo Wang (fungo)]

I recently encountered this same bug using percona-xtrabackup 2.2.9 to prepare

the crash info is as follow

2015-05-12 19:03:08 7fd9d8258720 InnoDB: Assertion failure in thread 140573610968864 in file pars0pars.cc line 865
InnoDB: Failing assertion: sym_node->table != NULL
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
11:03:08 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x176aeb0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
xtrabackup(my_print_stacktrace+0x35) [0x9f5331]
xtrabackup(handle_fatal_signal+0x2bb) [0x7f801b]
/lib64/libpthread.so.0() [0x3530c0f500]
/lib64/libc.so.6(gsignal+0x35) [0x35f40328a5]
/lib64/libc.so.6(abort+0x175) [0x35f4034085]
xtrabackup() [0x76bfb0]
xtrabackup(pars_update_statement(upd_node_t*, sym_node_t*, void*)+0x30) [0x76c8d8]
xtrabackup(yyparse()+0xcb1) [0xa5ef27]
xtrabackup(pars_sql(pars_info_t*, char const*)+0xaf) [0x76e06d]
xtrabackup(que_eval_sql(pars_info_t*, char const*, unsigned long, trx_t*)+0x85) [0x78eeb2]
xtrabackup(row_drop_table_for_mysql(char const*, trx_t*, bool, bool)+0xa98) [0x720a0c]
xtrabackup(row_mysql_drop_temp_tables()+0x24c) [0x721503]
xtrabackup(recv_recovery_rollback_active()+0x2c) [0x753ebe]
xtrabackup(innobase_start_or_create_for_mysql()+0x17aa) [0x7293c4]
xtrabackup() [0x607a00]
xtrabackup() [0x610204]
xtrabackup(main+0x8b8) [0x611674]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x35f401ecdd]
xtrabackup() [0x604369]

And this the repeat process is as follow:
1. a 5.5 server
2. establish a session to this sever, execute create temporary table tmp_crash(id int) engine=InnoDB, and hold this session so that tmp table don't get droped
3. do a backup against this server using xtrabackup 2.2
4. after backup finished, prepare using xtrabackup 2.2
and will see this crash.

After some dig into the code, it can be confirmed a incompatibility problem between MySQL 5.5(include PS55) and MySQL 5.6.

In Xtrabackup 2.2, the xtrabackup binary is compile against MySQL 5.6.22, and just one binary file, not like before there were multi version xtrabackup binary, compiled against MySQL 5.5/5.1 etc.
In MySQL 5.6 InnoDB, there are 2 more internal tables(SYS_TABLESPACES,SYS_DATAFILES) than 5.5, and that's the problem. If there are tmp table in backup file, where prepared, a InnoDB engine
started, and at start it will try to drop tmp table, which will lead to this crash. Cause in 5.5 there are no SYS_TABLESPACES and SYS...

[Sergei Glushchenko (sergei.glushchenko)]

Confirmed with this test case

############################################################################
# Bug 1399471: XtraBackup crash on prepare
# (when using --databases during backup)
############################################################################
. inc/common.sh

require_server_version_lower_than 5.6.0

start_server --innodb_file_per_table
load_sakila

function create_tmp_table()
{
    run_cmd $MYSQL $MYSQL_ARGS sakila <<EOF
START TRANSACTION;
CREATE TEMPORARY TABLE tmp_crash(id INT) ENGINE=InnoDB;
SELECT SLEEP(10000);
EOF
}

create_tmp_table &
job_id=$!

vlog "Starting backup"

innobackupex --no-timestamp $topdir/backup --include=sakila.payment

kill -SIGKILL $job_id

vlog "Preparing backup"
innobackupex --apply-log $topdir/backup

stop_server

innobackupex --copy-back $topdir/backup

start_server

[Sergei Glushchenko (sergei.glushchenko)]

Actually --include is not necessary to reproduce the crash and stack trace is identical to what we have in bug 1383062. Marking later as duplicate of this one and changing description.

[Sergei Glushchenko (sergei.glushchenko)]

https://github.com/percona/percona-xtrabackup/pull/63
https://github.com/percona/percona-xtrabackup/pull/64

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-417