handle_fatal_signal (sig=11) in hp_rec_hashnr on various queries

[Roel Van de Paar]

2013-09-16 04:28:21 17041 [Note] /ssd/Percona-Server-5.6.13-rc60.5-435-debug.Linux.x86_64/bin/mysqld: ready for connections.
Version: '5.6.13-rc60.5-debug-log' socket: '/ssd//759308/current1_3/tmp/master.sock' port: 13040 Percona Server with XtraDB (GPL), Release rc60.5, Revision 435-debug
2013-09-16 04:28:32 17041 [Warning] Could not remove temporary table: '/ssd/759308/current1_3/tmp/#sql4291_f_0', error: 121
01:28:45 UTC - mysqld got signal 11 ;
[...]
Query (7f4668004c50): INSERT INTO `view_table100_memory_int` ( `c16` ) VALUES ( -4627712 ) ON DUPLICATE KEY UPDATE `c14` = 1
Connection ID (thread ID): 15
Status: NOT_KILLED

=========

Thread 1 (LWP 17451):
+bt
#0 0x000000363720c69c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006874ae in handle_fatal_signal (sig=11) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/signal_handler.cc:251
#2 <signal handler called>
#3 hp_rec_hashnr (keydef=<optimized out>, rec=0x5c41 <Address 0x5c41 out of bounds>) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/heap/hp_hash.c:364
#4 0x00000000009f37f8 in check_one_key (print_status=0 '\000', blength=2, records=1, keynr=0, keydef=0x7f46540fccb0) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/heap/_check.c:146
#5 heap_check_heap (info=0x7f4668017200, print_status=0 '\000') at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/heap/_check.c:56
#6 0x00000000009f25ca in hp_close (info=0x7f4668017200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/heap/hp_close.c:39
#7 0x00000000009f26ad in heap_close (info=0x7f4668017200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/heap/hp_close.c:28
#8 0x00000000005b519d in handler::ha_close (this=0x7f466807f2a0) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/handler.cc:2576
#9 0x00000000007c44a9 in closefrm (table=0x7f46680bddd0, free_share=true) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/table.cc:2436
#10 0x00000000006c3657 in intern_close_table (table=0x7f46680bddd0) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_base.cc:821
#11 0x00000000006c9605 in close_thread_table (thd=<optimized out>, table_ptr=<optimized out>) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_base.cc:1525
#12 0x00000000006ca273 in close_open_tables (thd=0x2193200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_base.cc:1241
#13 close_thread_tables (thd=0x2193200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_base.cc:1467
#14 0x0000000000721c71 in mysql_execute_command (thd=0x2193200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_parse.cc:5228
#15 0x0000000000728628 in mysql_parse (thd=0x2193200, rawbuf=0x21955c0 "P\303\070\001", length=<optimized out>, parser_state=<optimized out>) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_parse.cc:6479
#16 0x000000000072978c in dispatch_command (command=COM_QUERY, thd=0x2193200, packet=0x7f46bf215a90 "", packet_length=104) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_parse.cc:1355
#17 0x000000000072a987 in do_command (thd=0x2193200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_parse.cc:1049
#18 0x00000000006ee1ef in do_handle_one_connection (thd_arg=<optimized out>) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_connect.cc:1615
#19 0x00000000006ee2d7 in handle_one_connection (arg=0x2193200) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/sql/sql_connect.cc:1526
#20 0x00000000009e033a in pfs_spawn_thread (arg=0x20af5e0) at /bzr/5.6-QA_dbg/Percona-Server-5.6.13-rc60.5/storage/perfschema/pfs.cc:1858
#21 0x0000003637207851 in start_thread () from /lib64/libpthread.so.0
#22 0x0000003636ee890d in clone () from /lib64/libc.so.6

[Roel Van de Paar (roel11)]

RQG @ 909 <email address hidden>
PS @ Percona-Server-5.6.13-rc60.5-435-debug.Linux.x86_64 (5.6-QA tree)

[Roel Van de Paar (roel11)]

Also see for FLUSH TABLES, adding occurence

Query (7f79f8004c50): FLUSH TABLES
Connection ID (thread ID): 11
Status: NOT_KILLED

[Laurynas Biveinis (laurynas-biveinis)]

Trashed ptr_to_rec in a found hash record. Either upstream, either caused by the dynamic row format patch.

Roel, can you Valgrind this one?

[Roel Van de Paar (roel11)]

Laurynas, tried, but the same trial run does not reproduce the bug. Is core analysis conducive?

Still trying with a longer duration run also.

[Roel Van de Paar (roel11)]

Hmm, I may also be able to run this under Valgrind, even if the trial does not reproduce the crash. The benefit may be dual (or none); Valgrind binary always tend to crash easier/earlier + the issue may be seen in Valgrind even if no crash is seen. Trying.

[Roel Van de Paar (roel11)]

Ok, many attempts have not resulted in anything else but STATUS_OK for valgrind runs or STATUS_SERVER_CRASHED or STATUS_OK for non-valgrind runs, so regrettably nothing usable in this way.

[Roel Van de Paar (roel11)]

Testcase reduction is also ~impossible here, or would take a full week or more. Laurynas, any bright ideas?

[Roel Van de Paar (roel11)]

May also be in 5.5 (see in 5.5 but no bt yet)

[Laurynas Biveinis (laurynas-biveinis)]

Roel, Ramesh, what is the reproduction procedure?

[Ramesh Sivaraman (rameshvs02)]

This is also seen in 5.5. Got the issue with many trails

[Ramesh Sivaraman (rameshvs02)]

Hi Laurynas,

We have used similar grammar file (using for 5.6 RQG) to run the 5.5 RQG. After analyzing the crash files, got handle_fatal_signal (sig=11) in hp_rec_hashnr function in many trails.

No full reproducible testcase yet, hope the backtraces reveal something

[Roel Van de Paar (roel11)]

Laurynas, can you have a look at the added stacks if they show what is needed?

[Laurynas Biveinis (laurynas-biveinis)]

No. Still need a reproducible test case.

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.5 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.6 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona Server because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-722