Percona Server crashes on checksum statement after table import

[Valentine Gostev]

Test case xb_export for Percona XtraBackup causes Percona Server 5.1 crash on checksum table statement.

Reproduced on Ubuntu 10.04 x86_64 w/
5.1.54-12.5
5.1.51-11.5

Backtrace from test log:

110126 2:30:32 InnoDB: Assertion failure in thread 140399065376512 in file row/row0sel.c line 3861
InnoDB: Failing assertion: rec
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110126 2:30:32 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338400 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x17edf30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb1346b1ca8 thread_stack 0x40000
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(my_print_stacktrace+0x35)[0x88e525]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(handle_segfault+0x31d)[0x5cb68d]
/lib/libpthread.so.0(+0xf8f0)[0x7fb146f358f0]
/lib/libc.so.6(gsignal+0x35)[0x7fb1462fca75]
/lib/libc.so.6(abort+0x180)[0x7fb1463005c0]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(row_search_for_mysql+0x97b)[0x7a441b]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x151)[0x745141]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase11index_firstEPh+0x37)[0x745847]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase8rnd_nextEPh+0x37)[0x745987]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z20mysql_checksum_tableP3THDP10TABLE_LISTP15st_ha_check_opt+0x348)[0x6bd098]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x33a9)[0x5de239]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x2b9)[0x5e1249]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x54a)[0x5e195a]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z10do_commandP3THD+0x11a)[0x5e2a8a]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(handle_one_connection+0x60b)[0x5d541b]
/lib/libpthread.so.0(+0x69ca)[0x7fb146f2c9ca]
/lib/libc.so.6(clone+0x6d)[0x7fb1463af70d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1845980 = checksum table test
thd->thread_id=6
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
t/xb_export.sh: line 48: 922 Aborted (core dumped) ${MYSQLD} ${MYSQLD_ARGS} $* --port=$mysql_port

Please find core dump at http://x54.in/core.922.gz

[Yasufumi Kinoshita (yasufumi-kinoshita)]

I don't know "xb_report".
I cannot do anything for now.

Please assign to the author of the "xb_report"
or add information for other developers to solve if you want to be solved by others.

[Valentine Gostev (longbow)]

I have linked a branch which contains xb_export test.

Download branch, change directory to test (inside branch content) and issue command:
~# ./run.sh -m percona -t t/xb_report.sh
in reports/xb_export.sh.out file you will find test log, containing backtrace.

[Yasufumi Kinoshita (yasufumi-kinoshita)]

You are wrong as I commented to the review.

[Vadim Tkachenko (vadim-tk)]

As we agreed, Percona Server should not crash.
We at least should check if ".exp" file exist.

I put bug into "confirmed", and I ask to prepare test case which crashes server without ".exp" file.

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Then, the code was already pushed.
It doesn't cause crash, when ".exp" file doesn't exist, when "innodb_expand_import" is enabled.
>> fprintf(stderr, "InnoDB: cannot open %s\n", info_file_path);

https://code.launchpad.net/~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit/+merge/48102

[Yasufumi Kinoshita (yasufumi-kinoshita)]

The fix is the part.

237 === modified file 'innodb_expand_import.patch'
238 --- innodb_expand_import.patch 2010-12-16 11:35:26 +0000
239 +++ innodb_expand_import.patch 2011-02-01 02:13:15 +0000
240 @@ -34,8 +34,8 @@
241 space_id = fsp_header_get_space_id(page);
242 space_flags = fsp_header_get_flags(page);
243
244 -+ if (srv_expand_import
245 -+ && (space_id != id || space_flags != (flags & ~(~0 << DICT_TF_BITS)))) {
246 ++ if (srv_expand_import) {
247 ++
248 + ibool file_is_corrupt = FALSE;
249 + byte* buf3;
250 + byte* descr_page;

[Vadim Tkachenko (vadim-tk)]

did you push to 5.5 ?

[Yasufumi Kinoshita (yasufumi-kinoshita)]

Sure.
http://bazaar.launchpad.net/~percona-dev/percona-server/5.5.8/revision/63

[Peter Zaitsev (pz-percona)]

Valentine,

Can you please ensure to post for every release in which bug was present an exact version which contains a fix.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-459