handle_fatal_signal (sig=11) in free (from my_free | ha_innobase::close ) | handle_fatal_signal (sig=11) in handler::ha_close

[Roel Van de Paar]

Thread 1 (LWP 14017):
+bt
#0 0x000000340cc0c69c in pthread_kill () from /lib64/libpthread.so.0
#1 0x0000000000686bde in handle_fatal_signal (sig=11) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/signal_handler.cc:251
#2 <signal handler called>
#3 0x000000340c47b6ec in free () from /lib64/libc.so.6
#4 0x0000000000983ba6 in my_free (ptr=0x8f8f8f8f8f8f8f8f) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/mysys/my_malloc.c:137
#5 0x00000000009ffd6f in ha_innobase::close (this=0x7f1e14037320) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/storage/innobase/handler/ha_innodb.cc:5387
#6 0x00000000005b48cd in handler::ha_close (this=0x7f1e14037320) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/handler.cc:2550
#7 0x0000000000755653 in store_temporary_table_record (thd=0x33b7390, table=0x7f1e1400a9e0, tmp_table=<optimized out>, db=<optimized out>) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_show.cc:3990
#8 0x0000000000764b39 in fill_global_temporary_tables (thd=0x35a01d0, tables=0x7f1e14005010, cond=<optimized out>) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_show.cc:4043
#9 0x0000000000759219 in do_fill_table (join_table=<optimized out>, table_list=0x7f1e14005010, thd=0x35a01d0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_show.cc:7714
#10 get_schema_tables_result (join=0x7f1e140067e8, executed_place=PROCESSED_BY_JOIN_EXEC) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_show.cc:7815
#11 0x0000000000746425 in JOIN::prepare_result (this=0x7f1e140067e8, columns_list=<optimized out>) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_select.cc:823
#12 0x00000000006fd211 in JOIN::exec (this=0x7f1e140067e8) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_executor.cc:116
#13 0x00000000007492d0 in mysql_execute_select (free_join=true, select_lex=0x35a2ca8, thd=0x35a01d0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_select.cc:1101
#14 mysql_select (thd=0x35a01d0, tables=0x7f1e14005010, wild_num=1, fields=..., conds=0x7f1e14005d78, order=<optimized out>, group=0x35a2da8, having=0x0, select_options=2684619520, result=0x7f1e140067c0, unit=0x35a2660, select_lex=0x35a2ca8) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_select.cc:1222
#15 0x0000000000749dbb in handle_select (thd=0x35a01d0, result=0x7f1e140067c0, setup_tables_done_option=0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_select.cc:111
#16 0x000000000071ecdd in execute_sqlcom_select (thd=0x35a01d0, all_tables=0x7f1e14005010) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:5305
#17 0x00000000007217b7 in mysql_execute_command (thd=0x35a01d0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:2762
#18 0x0000000000727818 in mysql_parse (thd=0x35a01d0, rawbuf=0x35a2598 "\360R7\001", length=<optimized out>, parser_state=<optimized out>) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:6477
#19 0x000000000072871f in dispatch_command (command=COM_QUERY, thd=0x35a01d0, packet=0x35a4161 " SELECT * FROM INFORMATION_SCHEMA . `GLOBAL_TEMPORARY_TABLES` AS table1 WHERE ( table1 . `ENGINE` <> table1 . `SESSION_ID` OR table1 . `DATA_LENGTH` >= '2003-01-07 09:45:33.021131') ", packet_length=182) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:1374
#20 0x0000000000729a47 in do_command (thd=0x35a01d0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:1049
#21 0x00000000006ed63f in do_handle_one_connection (thd_arg=<optimized out>) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_connect.cc:1615
#22 0x00000000006ed727 in handle_one_connection (arg=0x35a01d0) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/sql/sql_connect.cc:1526
#23 0x00000000009dd6c9 in pfs_spawn_thread (arg=0x3526e90) at /bzr/5.6_dbg/Percona-Server-5.6.12-rc60.4/storage/perfschema/pfs.cc:1855
#24 0x000000340cc07851 in start_thread () from /lib64/libpthread.so.0
#25 0x000000340c4e890d in clone () from /lib64/libc.so.6

Query (7f1e14004c50): SELECT * FROM INFORMATION_SCHEMA . `GLOBAL_TEMPORARY_TABLES` AS table1 WHERE ( table1 . `ENGINE` <> table1 . `SESSION_ID` OR table1 . `DATA_LENGTH` >= '2003-01-07 09:45:33.021131')

[Roel Van de Paar (roel11)]

The same cause as bug 1193308 maybe?

[Roel Van de Paar (roel11)]

ps -ef | grep 'cmdrun_484' | grep -v grep | awk '{print $2}' | xargs sudo kill -9
rm -Rf /ssd/209184/cmdrun_484
mkdir /ssd/209184/cmdrun_484
cd /ssd/randgen
bash -c "set -o pipefail; perl /ssd/randgen/runall.pl --queries=100000000 --seed=28484 --duration=300 --querytimeout=60 --reporter=Shutdown,Backtrace,QueryTimeout,ErrorLog,ErrorLogAlarm --mysqld=--log-output=none --mysqld=--sql_mode=ONLY_FULL_GROUP_BY --mysqld=--slow_query_log --mysqld=--userstat --threads=25 --basedir=/ssd/Percona-Server-5.6.12-rc60.4-405-debug.Linux.x86_64 --grammar=/tmp/976291/4.yy --gendata=/ssd/randgen/conf/replication/replication-6.0.zz --mtr-build-thread=791 --mask=50980 --vardir1=/ssd/209184/cmdrun_484 > /ssd/209184/cmdrun484.log 2>&1"

[Laurynas Biveinis (laurynas-biveinis)]

The problem is thrashed upd_buf pointer field in the cloned handler. I am unable to tell how it became thrashed because this field is not copied from the original, it should be constant NULL.

It's possible to paper over it by not freeing upd_buf in the cloned handlers (this could be the proper fix too once the cause is known), or maybe pepper handler with asserts that upd_buf == NULL if cloned handler, but maybe let's leave it as-is and hope to get more crashes from RQG.

[Roel Van de Paar (roel11)]

New occurrence in first iteration of 5.6 GA tree

[Roel Van de Paar (roel11)]

[roel@qaserver 873186]$ cat /ssd/randgen/.bzr/branch/last-revision
894 <email address hidden>

[Roel Van de Paar (roel11)]

[roel@qaserver randgen]$ cat .bzr/branch/last-revision
897 <email address hidden>

[Alexey Kopytov (akopytov)]

Not a duplicate of bug #1217213.

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.5 because there has been no activity for 60 days.]

[Launchpad Janitor (janitor)]

[Expired for Percona Server 5.6 because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-698