xtrabackup crashed during apply 5.5 backup set with utf8_general50_ci collation

[Fungo Wang]

When I recover backup set from PS 5.5 using PXB 2.2/2.3, xtrabackup crashed:

2016-01-13 22:59:45 2ad8f06cf480 InnoDB: Assertion failure in thread 47111234974848 in file ha_innodb.cc line 1615
InnoDB: Failing assertion: cset == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
14:59:45 UTC - xtrabackup got signal 6 ;
This could be because you hit a bug or data is corrupted.
This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x10000
innobackupex(my_print_stacktrace+0x2e) [0x93827e]
innobackupex(handle_fatal_signal+0x273) [0x81b483]
/lib64/libpthread.so.0(+0xf500) [0x2ad8ef862500]
/lib64/libc.so.6(gsignal+0x35) [0x2ad8f036d8a5]
/lib64/libc.so.6(abort+0x175) [0x2ad8f036f085]
innobackupex() [0x6894a2]
innobackupex(dict_mem_fill_column_struct(dict_col_t*, unsigned long, unsigned long, unsigned long, unsigned long)+0x9a) [0x5f79da]
innobackupex(dict_load_column_low(dict_table_t*, mem_block_info_t*, dict_col_t*, unsigned long*, char const**, unsigned char const*)+0x3be) [0x611b2e]
innobackupex(dict_load_table(char const*, unsigned long, dict_err_ignore_t)+0x5f2) [0x612ae2]
innobackupex(dict_load_table_on_id(unsigned long, dict_err_ignore_t)+0x4d0) [0x617980]
innobackupex(dict_table_open_on_id(unsigned long, unsigned long, dict_table_op_t)+0xf8) [0x62a698]
innobackupex() [0x5c34e6]
innobackupex(trx_lists_init_at_db_start()+0x2be) [0x5c393e]
innobackupex(trx_sys_init_at_db_start()+0x182) [0x5d6b32]
innobackupex(innobase_start_or_create_for_mysql()+0x1736) [0x6d7646]
innobackupex() [0x58fac7]
innobackupex(main+0xaba) [0x591f8a]
/lib64/libc.so.6(__libc_start_main+0xfd) [0x2ad8f0359cdd]
innobackupex() [0x585ea9]

Please report a bug at https://bugs.launchpad.net/percona-xtrabackup
Writing a core file
Aborted

When use the binary complied from source, and run with gdb, the backtrace is

#0 0x00002aaaabbe48a5 in raise () from /lib64/libc.so.6
#1 0x00002aaaabbe6085 in abort () from /lib64/libc.so.6
#2 0x00000000006e9a7e in innobase_get_cset_width (cset=253, mbminlen=0x7fffffff52f8, mbmaxlen=0x7fffffff52f0) at percona-xtrabackup/storage/innobase/handler/ha_innodb.cc:1615
#3 0x00000000006a32f5 in dtype_get_mblen (mtype=13, prtype=16581118, mbminlen=0x7fffffff52f8, mbmaxlen=0x7fffffff52f0) at percona-xtrabackup/storage/innobase/include/data0type.ic:96
#4 0x00000000006a5118 in dict_mem_fill_column_struct (column=0x1a3d250, col_pos=2, mtype=13, prtype=16581118, col_len=360) at percona-xtrabackup/storage/innobase/dict/dict0mem.cc:493
#5 0x00000000006a470b in dict_mem_table_add_col (table=0x1a3cfd8, heap=0x1a350f0, name=0x1a35230 "c", mtype=13, prtype=16581118, len=360) at percona-xtrabackup/storage/innobase/dict/dict0mem.cc:262
#6 0x000000000075dc87 in dict_load_column_low (table=0x1a3cfd8, heap=0x1a350f0, column=0x0, table_id=0x0, col_name=0x7fffffff59f0, rec=0x2aaaac8584fc "") at percona-xtrabackup/storage/innobase/dict/dict0load.cc:1361
#7 0x000000000075df80 in dict_load_columns (table=0x1a3cfd8, heap=0x1a350f0) at percona-xtrabackup/storage/innobase/dict/dict0load.cc:1423
#8 0x00000000007605b9 in dict_load_table (name=0x1a35038 "test/sbtest1", cached=1, ignore_err=DICT_ERR_IGNORE_RECOVER_LOCK) at percona-xtrabackup/storage/innobase/dict/dict0load.cc:2453
#9 0x0000000000760c9e in dict_load_table_on_id (table_id=16, ignore_err=DICT_ERR_IGNORE_RECOVER_LOCK) at percona-xtrabackup/storage/innobase/dict/dict0load.cc:2661
#10 0x00000000007f2123 in dict_table_open_on_id_low (table_id=16, ignore_err=DICT_ERR_IGNORE_RECOVER_LOCK) at percona-xtrabackup/storage/innobase/include/dict0priv.ic:92
#11 0x00000000007f35f9 in dict_table_open_on_id (table_id=16, dict_locked=0, table_op=DICT_TABLE_OP_LOAD_TABLESPACE) at percona-xtrabackup/storage/innobase/dict/dict0dict.cc:945
#12 0x000000000088bf31 in trx_resurrect_table_locks (trx=0x1a340f8, undo=0x1a1fd68) at percona-xtrabackup/storage/innobase/trx/trx0trx.cc:473
#13 0x000000000088c743 in trx_lists_init_at_db_start () at percona-xtrabackup/storage/innobase/trx/trx0trx.cc:744
#14 0x00000000008070b9 in trx_sys_init_at_db_start () at percona-xtrabackup/storage/innobase/trx/trx0sys.cc:579
#15 0x00000000006aaad0 in innobase_start_or_create_for_mysql () at percona-xtrabackup/storage/innobase/srv/srv0start.cc:2435
#16 0x000000000061779d in innodb_init () at percona-xtrabackup/storage/innobase/xtrabackup/src/xtrabackup.cc:1799
#17 0x00000000006200ac in xtrabackup_prepare_func () at percona-xtrabackup/storage/innobase/xtrabackup/src/xtrabackup.cc:6267
#18 0x0000000000621883 in main (argc=133, argv=0x1718898) at percona-xtrabackup/storage/innobase/xtrabackup/src/xtrabackup.cc:7001

With the backtrace, and after reading the source code, I found it is a compatibility problem between PS 5.5 and Oracle/MySQL 5.6.

As we can see from the backtrace, cset=253, corresponding to 'utf8_general50_ci' collation in PS 5.5, which not exists in Oracle/MySQL 5.6, and xtrabackup 2.2/2.3 is compiled against Oracle/MySQL 5.6.

After some google dig, I found a related bug report for Percona Server here https://bugs.launchpad.net/percona-server/+bug/1163324

In order to fix some early bugs, PS introduced two collation utf8_general50_ci and ucs2_general50_ci, but Oracle use the other two collations utf8_general_mysql500_ci and ucs2_general_mysql500_ci.

ucs2_general_mysql500_ci(id=159) is compatible with ucs2_general50_ci(id=159), but utf8_general_mysql500_ci(id=223) is not compatible with utf8_general50_ci(id=253), so the crash.

I think a simple fix would be allow 253 collaton in innobase_get_cset_width(), just print warnings instead crash. Attachment is proposed fix patch.

[Fungo Wang (fungo)]

BTW, such backup set can be constructed by creating table like this:

CREATE TABLE `sbtest1` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `k` int(10) unsigned NOT NULL DEFAULT '0',
  `c` char(120) CHARACTER SET utf8 COLLATE utf8_general50_ci NOT NULL DEFAULT '',
  `pad` char(60) NOT NULL DEFAULT '',
  PRIMARY KEY (`id`),
  KEY `k_1` (`k`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

and then take a backup.
During backup, please update this table sbtest1 frequently, such as using sysbench.

[Vasily Nemkov (vasily.nemkov)]

PR: https://github.com/percona/percona-xtrabackup/pull/435

[Vasily Nemkov (vasily.nemkov)]

PR: https://github.com/percona/percona-xtrabackup/pull/438

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXB-748