handle_fatal_signal (sig=11) in my_lfind

[Roel Van de Paar]

This is a multi-threaded bug. It was reduced down to this small testcase;

# mysqld options required for replay: --maximum-tmp_table_size=1M --maximum-transaction_prealloc_size=1M --sql_mode=ONLY_FULL_GROUP_BY --thread_handling=pool-of-threads --log-bin=binlog --plugin-load-add=tokudb=ha_tokudb.so --init-file=/home/roel/percona-qa/plugins_56.sql --tokudb-check-jemalloc=0 --performance-schema-events-waits-history-size=0 --tokudb-last-lock-timeout=4
DROP DATABASE transforms;
CREATE TABLE t1(id INT,a BLOB COLUMN_FORMAT COMPRESSED WITH COMPRESSION_DICTIONARY numbers)ENGINE=TokuDB;

And it then needs a multi-thread wrapper (like the one in percona-qa perhaps, or reducer) to run it

Laurynas analyzed the stacks and the issue looks to be a race in threadpool.

Thread 0:

Core was generated by `/sda/COMP7-PS111016-percona-server-5.6.32-78.0-linux-x86_64-debug/bin/mysqld --'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000000000e4d32d in my_lfind (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336LJ\255]\r'K8\001(\372test", keylen=212, cursor=0x7f6e56c5dfb0, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:95
95 link= cursor->curr->link;
(gdb) bt
+bt
#0 0x0000000000e4d32d in my_lfind (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336LJ\255]\r'K8\001(\372test", keylen=212, cursor=0x7f6e56c5dfb0, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:95
#1 0x0000000000e4d7ab in my_lsearch (head=0x7f6e2c438e28, cs=0x14e9260 <my_charset_bin>, hashnr=674812401, key=0x7f6e56c5e050 "\331\061\242$\336LJ\255]\r'K8\001(\372test", keylen=212, pins=0x7f6e2d816300) at /git/compression7_dbg/mysys/lf_hash.c:267
#2 0x0000000000e4de4e in lf_hash_search (hash=0x17d5780 <digest_hash>, pins=0x7f6e2d816300, key=0x7f6e56c5e050, keylen=212) at /git/compression7_dbg/mysys/lf_hash.c:462
#3 0x0000000000ae1956 in find_or_create_digest (thread=0x7f6e6b9d5700, digest_storage=0x7f6e4ffc4c60, schema_name=0x7f6e4ffc4d48 "test", schema_name_length=4) at /git/compression7_dbg/storage/perfschema/pfs_digest.cc:223
#4 0x0000000000b1696d in end_statement_v1 (locker=0x7f6e4ffc4ca0, stmt_da=0x7f6e4ffc64a0) at /git/compression7_dbg/storage/perfschema/pfs.cc:4815
#5 0x00000000007e3239 in inline_mysql_end_statement (locker=0x7f6e4ffc4ca0, stmt_da=0x7f6e4ffc64a0) at /git/compression7_dbg/include/mysql/psi/mysql_statement.h:215
#6 0x00000000007e8065 in dispatch_command (command=COM_QUERY, thd=0x7f6e4ffc3000, packet=0x7f6e4ffc7001 "", packet_length=87) at /git/compression7_dbg/sql/sql_parse.cc:1911
#7 0x00000000007e595d in do_command (thd=0x7f6e4ffc3000) at /git/compression7_dbg/sql/sql_parse.cc:1062
#8 0x00000000008cbba0 in threadpool_process_request (thd=0x7f6e4ffc3000) at /git/compression7_dbg/sql/threadpool_common.cc:314
#9 0x00000000008ce48a in handle_event (connection=0x7f6e4fef6bb0) at /git/compression7_dbg/sql/threadpool_unix.cc:1564
#10 0x00000000008ce6e4 in worker_main (param=0x17bdc00 <all_groups+512>) at /git/compression7_dbg/sql/threadpool_unix.cc:1617
#11 0x0000000000b10701 in pfs_spawn_thread (arg=0x7f6e2a084020) at /git/compression7_dbg/storage/perfschema/pfs.cc:1860
#12 0x00007f6e8078edc5 in start_thread (arg=0x7f6e56c5f700) at pthread_create.c:308
#13 0x00007f6e7ebebced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Signal thread:

Thread 69 (Thread 0x7f6e5fc6b700 (LWP 4988)):
+bt
#0 0x00007f6e7eec86e3 in _Unwind_IteratePhdrCallback (info=info@entry=0x7f6e5fc69050, size=size@entry=64, ptr=ptr@entry=0x7f6e5fc690d0) at ../../../libgcc/unwind-dw2-fde-dip.c:398
#1 0x00007f6e7ec25fac in __GI___dl_iterate_phdr (callback=callback@entry=0x7f6e7eec82e0 <_Unwind_IteratePhdrCallback>, data=data@entry=0x7f6e5fc690d0) at dl-iteratephdr.c:76
#2 0x00007f6e7eec8c1f in _Unwind_Find_FDE (pc=0x7f6e7ebffbe5 <__GI___backtrace+85>, bases=bases@entry=0x7f6e5fc69268) at ../../../libgcc/unwind-dw2-fde-dip.c:461
#3 0x00007f6e7eec5d8c in uw_frame_state_for (context=context@entry=0x7f6e5fc691c0, fs=fs@entry=0x7f6e5fc692b0) at ../../../libgcc/unwind-dw2.c:1245
#4 0x00007f6e7eec7019 in _Unwind_Backtrace (trace=0x7f6e7ebffa70 <backtrace_helper>, trace_argument=0x7f6e5fc69470) at ../../../libgcc/unwind.inc:290
#5 0x00007f6e7ebffbe6 in __GI___backtrace (array=<optimized out>, size=128) at ../sysdeps/x86_64/backtrace.c:109
#6 0x0000000000ac1b34 in my_print_stacktrace (stack_bottom=0x7f6e5fc6ac08 "", thread_stack=262144) at /git/compression7_dbg/mysys/stacktrace.c:224
#7 0x00000000007361e6 in handle_fatal_signal (sig=11) at /git/compression7_dbg/sql/signal_handler.cc:163
#8 <signal handler called>
#9 0x0000000000e4d32d in my_lfind (head=0x7f6e2d83f528, cs=0x14e9260 <my_charset_bin>, hashnr=694946247, key=0x7f6e5fc6a050 "O(r\332\353\317m\352\207ڨ\032u\t\324\354compressed_db", keylen=212, cursor=0x7f6e5fc69fb0, pins=0x7f6e2d816280) at /git/compression7_dbg/mysys/lf_hash.c:95
#10 0x0000000000e4d7ab in my_lsearch (head=0x7f6e2d83f528, cs=0x14e9260 <my_charset_bin>, hashnr=694946247, key=0x7f6e5fc6a050 "O(r\332\353\317m\352\207ڨ\032u\t\324\354compressed_db", keylen=212, pins=0x7f6e2d816280) at /git/compression7_dbg/mysys/lf_hash.c:267
#11 0x0000000000e4de4e in lf_hash_search (hash=0x17d5780 <digest_hash>, pins=0x7f6e2d816280, key=0x7f6e5fc6a050, keylen=212) at /git/compression7_dbg/mysys/lf_hash.c:462
#12 0x0000000000ae1956 in find_or_create_digest (thread=0x7f6e6b9d3600, digest_storage=0x7f6e4ffb2c60, schema_name=0x7f6e4ffb2d48 "compressed_db", schema_name_length=13) at /git/compression7_dbg/storage/perfschema/pfs_digest.cc:223
#13 0x0000000000b1696d in end_statement_v1 (locker=0x7f6e4ffb2ca0, stmt_da=0x7f6e4ffb44a0) at /git/compression7_dbg/storage/perfschema/pfs.cc:4815
#14 0x00000000007e3239 in inline_mysql_end_statement (locker=0x7f6e4ffb2ca0, stmt_da=0x7f6e4ffb44a0) at /git/compression7_dbg/include/mysql/psi/mysql_statement.h:215
#15 0x00000000007e8065 in dispatch_command (command=COM_QUERY, thd=0x7f6e4ffb1000, packet=0x7f6e4ffb5001 "", packet_length=65) at /git/compression7_dbg/sql/sql_parse.cc:1911
#16 0x00000000007e595d in do_command (thd=0x7f6e4ffb1000) at /git/compression7_dbg/sql/sql_parse.cc:1062
#17 0x00000000008cbba0 in threadpool_process_request (thd=0x7f6e4ffb1000) at /git/compression7_dbg/sql/threadpool_common.cc:314
#18 0x00000000008ce48a in handle_event (connection=0x7f6e4fef6af0) at /git/compression7_dbg/sql/threadpool_unix.cc:1564
#19 0x00000000008ce6e4 in worker_main (param=0x17be800 <all_groups+3584>) at /git/compression7_dbg/sql/threadpool_unix.cc:1617
#20 0x0000000000b10701 in pfs_spawn_thread (arg=0x7f6e2d01b0c0) at /git/compression7_dbg/storage/perfschema/pfs.cc:1860
#21 0x00007f6e8078edc5 in start_thread (arg=0x7f6e5fc6b700) at pthread_create.c:308
#22 0x00007f6e7ebebced in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Upon entering gdb;

(gdb) p cursor
+p cursor
$1 = (CURSOR *) 0x7f6e56c5dfb0

(gdb) p cursor->curr
+p cursor->curr
$2 = (LF_SLIST *) 0x2f62645f64657372

All details will be uploaded.

[Roel Van de Paar (roel11)]

Adding core dump + mysqld + ldd files + full data dir and original thread sql files etc.

To init gdb:

gdb ./mysqld/mysqld ./data/core.4840.1000.1000.11.1476222798.mysqld

1037.sql_out_out is the final testcase (listed above)

Even has reducer1037.sh included which can be path-modified to reproduce in-spe

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3578