Percona Server with XtraDB

Percona Server crashes on checksum statement after table import

Reported by Valentine Gostev on 2011-01-26
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server
High
Unassigned

Bug Description

Test case xb_export for Percona XtraBackup causes Percona Server 5.1 crash on checksum table statement.

Reproduced on Ubuntu 10.04 x86_64 w/
5.1.54-12.5
5.1.51-11.5

Backtrace from test log:

110126 2:30:32 InnoDB: Assertion failure in thread 140399065376512 in file row/row0sel.c line 3861
InnoDB: Failing assertion: rec
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.1/en/forcing-recovery.html
InnoDB: about forcing recovery.
110126 2:30:32 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=1
max_threads=151
threads_connected=1
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338400 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

thd: 0x17edf30
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7fb1346b1ca8 thread_stack 0x40000
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(my_print_stacktrace+0x35)[0x88e525]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(handle_segfault+0x31d)[0x5cb68d]
/lib/libpthread.so.0(+0xf8f0)[0x7fb146f358f0]
/lib/libc.so.6(gsignal+0x35)[0x7fb1462fca75]
/lib/libc.so.6(abort+0x180)[0x7fb1463005c0]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(row_search_for_mysql+0x97b)[0x7a441b]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase10index_readEPhPKhj16ha_rkey_function+0x151)[0x745141]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase11index_firstEPh+0x37)[0x745847]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_ZN11ha_innobase8rnd_nextEPh+0x37)[0x745987]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z20mysql_checksum_tableP3THDP10TABLE_LISTP15st_ha_check_opt+0x348)[0x6bd098]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z21mysql_execute_commandP3THD+0x33a9)[0x5de239]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z11mysql_parseP3THDPcjPPKc+0x2b9)[0x5e1249]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x54a)[0x5e195a]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(_Z10do_commandP3THD+0x11a)[0x5e2a8a]
/root/xb_export_test/test/var/Percona-Server-5.1.51-rel11.5-132-Linux-x86_64/libexec/mysqld(handle_one_connection+0x60b)[0x5d541b]
/lib/libpthread.so.0(+0x69ca)[0x7fb146f2c9ca]
/lib/libc.so.6(clone+0x6d)[0x7fb1463af70d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd->query at 0x1845980 = checksum table test
thd->thread_id=6
thd->killed=NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query
t/xb_export.sh: line 48: 922 Aborted (core dumped) ${MYSQLD} ${MYSQLD_ARGS} $* --port=$mysql_port

Please find core dump at http://x54.in/core.922.gz

Related branches

lp:~percona-dev/percona-xtrabackup/xb_export
Merged into lp:percona-xtrabackup/2.0 at revision 224
Vadim Tkachenko: Approve on 2011-02-10
Yasufumi Kinoshita: Pending requested 2011-02-01
Alexey Kopytov: Pending requested 2011-02-01
lp:~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit
Valentine Gostev (community): Approve (qa) on 2011-02-03
Fred Linhoss (community): Approve (documentation) on 2011-02-02
Percona developers: Pending requested 2011-02-01
Changed in percona-server:
milestone: none → release-5.1.54-12.5
importance: Undecided → High
status: New → Triaged
assignee: nobody → Yasufumi Kinoshita (yasufumi-kinoshita)

I don't know "xb_report".
I cannot do anything for now.

Please assign to the author of the "xb_report"
or add information for other developers to solve if you want to be solved by others.

Changed in percona-server:
assignee: Yasufumi Kinoshita (yasufumi-kinoshita) → nobody
description: updated
Valentine Gostev (longbow) wrote :

I have linked a branch which contains xb_export test.

Download branch, change directory to test (inside branch content) and issue command:
~# ./run.sh -m percona -t t/xb_report.sh
in reports/xb_export.sh.out file you will find test log, containing backtrace.

Changed in percona-server:
assignee: nobody → Valentine Gostev (core-longbow)
assignee: Valentine Gostev (core-longbow) → Yasufumi Kinoshita (yasufumi-kinoshita)

You are wrong as I commented to the review.

Changed in percona-server:
status: Triaged → Invalid
Vadim Tkachenko (vadim-tk) wrote :

As we agreed, Percona Server should not crash.
We at least should check if ".exp" file exist.

I put bug into "confirmed", and I ask to prepare test case which crashes server without ".exp" file.

Changed in percona-server:
status: Invalid → Confirmed

Then, the code was already pushed.
It doesn't cause crash, when ".exp" file doesn't exist, when "innodb_expand_import" is enabled.
>> fprintf(stderr, "InnoDB: cannot open %s\n", info_file_path);

https://code.launchpad.net/~percona-dev/percona-server/5.1.54-fix_expand_import_and_dict_size_limit/+merge/48102

Changed in percona-server:
status: Confirmed → Fix Committed

The fix is the part.

237 === modified file 'innodb_expand_import.patch'
238 --- innodb_expand_import.patch 2010-12-16 11:35:26 +0000
239 +++ innodb_expand_import.patch 2011-02-01 02:13:15 +0000
240 @@ -34,8 +34,8 @@
241 space_id = fsp_header_get_space_id(page);
242 space_flags = fsp_header_get_flags(page);
243
244 -+ if (srv_expand_import
245 -+ && (space_id != id || space_flags != (flags & ~(~0 << DICT_TF_BITS)))) {
246 ++ if (srv_expand_import) {
247 ++
248 + ibool file_is_corrupt = FALSE;
249 + byte* buf3;
250 + byte* descr_page;

Vadim Tkachenko (vadim-tk) wrote :

did you push to 5.5 ?

Changed in percona-server:
status: Fix Committed → Fix Released
Peter Zaitsev (pz-percona) wrote :

Valentine,

Can you please ensure to post for every release in which bug was present an exact version which contains a fix.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers