handle_fatal_signal (sig=11) in je_bitmap_set | include/jemalloc/internal/bitmap.h:140

Bug #1457749 reported by Ramesh Sivaraman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Status tracked in 5.7
5.1
Invalid
Undecided
Unassigned
5.5
Invalid
Undecided
Unassigned
5.6
Triaged
High
Unassigned
5.7
Triaged
High
Unassigned

Bug Description

Two different assertion with almost similar testcase.
Testcase difference is

DROP DATABASE transforms;CREATE DATABASE transforms;DROP DATABASE test;CREATE DATABASE test;USE test;

vs

DROP DATABASE test;CREATE DATABASE test;USE test;

1st one

** Testcase

DROP DATABASE test;CREATE DATABASE test;USE test;
UPDATE performance_schema.threads SET INSTRUMENTED='NO' WHERE PROCESSLIST_ID = CONNECTION_ID();
set tokudb_optimize_index_name='primary';
SET STATEMENT myisam_sort_buffer_size=320000, myisam_repair_threads=2, sort_buffer_size=220022, binlog_format=row, keep_files_on_create=ON, max_join_size=2222220000000 FOR SELECT @@myisam_sort_buffer_size, @@myisam_repair_threads, @@sort_buffer_size, @@binlog_format, @@keep_files_on_create, @@max_join_size;
set session profiling=ON;
set session profiling = OFF;
set tokudb_optimize_index_name='b';
CREATE TEMPORARY TABLE t1 AS SELECT thread_id FROM performance_schema.threads WHERE name LIKE 'thread/sql%';
show profile memory;

Error occurring while shutting down server

** Stack trace

+bt
#0 0x00007fc90ace4771 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000abe4d2 in my_write_core (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/mysys/stacktrace.c:422
#2 0x0000000000731fa3 in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/signal_handler.cc:236
#3 <signal handler called>
#4 __memset_x86_64 () at ../sysdeps/x86_64/memset.S:1316
#5 0x0000000000ab2677 in free_root (root=0x7fc8f7354420, MyFlags=1) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/mysys/my_alloc.c:372
#6 0x00000000007e3597 in dispatch_command (command=COM_QUERY, thd=0x7fc8f7351000, packet=0x7fc8f7355001 "", packet_length=30) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:1905
#7 0x00000000007e0e28 in do_command (thd=0x7fc8f7351000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:1054
#8 0x00000000007a8dad in do_handle_one_connection (thd_arg=0x7fc8f16ee000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_connect.cc:1532
#9 0x00000000007a88b5 in handle_one_connection (arg=0x7fc8f16ee000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_connect.cc:1443
#10 0x0000000000dcfb11 in pfs_spawn_thread (arg=0x7fc906fff7e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/perfschema/pfs.cc:1860
#11 0x00007fc90acdfdf5 in start_thread (arg=0x7fc90b297700) at pthread_create.c:308
#12 0x00007fc9099a91ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

2nd one

** Testcase

DROP DATABASE transforms;CREATE DATABASE transforms;DROP DATABASE test;CREATE DATABASE test;USE test;
UPDATE performance_schema.threads SET INSTRUMENTED='NO' WHERE PROCESSLIST_ID = CONNECTION_ID();
set tokudb_optimize_index_name='primary';
SET STATEMENT myisam_sort_buffer_size=320000, myisam_repair_threads=2, sort_buffer_size=220022, binlog_format=row, keep_files_on_create=ON, max_join_size=2222220000000 FOR SELECT @@myisam_sort_buffer_size, @@myisam_repair_threads, @@sort_buffer_size, @@binlog_format, @@keep_files_on_create, @@max_join_size;
set session profiling=ON;
set session profiling = OFF;
set tokudb_optimize_index_name='b';
CREATE TEMPORARY TABLE t1 AS SELECT thread_id FROM performance_schema.threads WHERE name LIKE 'thread/sql%';
show profile memory;

Need to execute twice to get the error

** Stack trace

#0 0x00007f187beed771 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000abe4d2 in my_write_core (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/m
ysys/stacktrace.c:422
#2 0x0000000000731fa3 in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-
72.2/sql/signal_handler.cc:236
#3 <signal handler called>
#4 je_bitmap_set (bit=18446744073709551553, binfo=0x7f187c32f958 <je_arena_bin_info+248>, bitmap=0x7f18018f1010) at include/jemalloc/internal/bitmap.h:105
#5 je_bitmap_sfu (binfo=0x7f187c32f958 <je_arena_bin_info+248>, bitmap=0x7f18018f1010) at include/jemalloc/internal/bitmap.h:140
#6 arena_run_reg_alloc (bin_info=0x7f187c32f930 <je_arena_bin_info+208>, run=0x7f18018f1000) at src/arena.c:291
#7 je_arena_tcache_fill_small (arena=0x7f18726c4e00, tbin=tbin@entry=0x7f1801806068, binind=binind@entry=2, prof_accumbytes=prof_accumbytes@entry=0) at src
/arena.c:1479
#8 0x00007f187c1236ff in je_tcache_alloc_small_hard (tcache=tcache@entry=0x7f1801806000, tbin=tbin@entry=0x7f1801806068, binind=binind@entry=2) at src/tcac
he.c:72
#9 0x00007f187c10314f in je_tcache_alloc_small (zero=false, size=32, tcache=0x7f1801806000) at include/jemalloc/internal/tcache.h:303
#10 je_arena_malloc (try_tcache=true, zero=false, size=<optimized out>, arena=0x0) at include/jemalloc/internal/arena.h:957
#11 je_imalloct (arena=0x0, try_tcache=true, size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:771
#12 je_imalloc (size=<optimized out>) at include/jemalloc/internal/jemalloc_internal.h:780
#13 malloc (size=<optimized out>) at src/jemalloc.c:929
#14 0x00007f1872d493fe in os_malloc (size=24) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/tokudb/ft-index/portability/os_malloc.cc:267
#15 0x00007f1872d46e90 in toku_xmalloc (size=24) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/tokudb/ft-index/portability/memory.cc:351
#16 0x00007f1872d18339 in toku_dbt_array_init (dbts=0x7f18018efe18, size=1) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/tokudb/ft-index/util/dbt.cc:119
#17 0x00007f1872c043af in ha_tokudb::ha_tokudb (this=0x7f18018ef010, hton=0x7f1875fe0a80, table_arg=0x0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/tokudb/ha_tokudb.cc:1226
#18 0x00007f1872c320a2 in tokudb_create_handler (hton=0x7f1875fe0a80, table=0x0, mem_root=0x7f1868754420) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/tokudb/hatoku_hton.cc:611
#19 0x000000000064d846 in exts_handlerton (unused=0x0, plugin=0x7f187c49beb8, arg=0x7f187c49bf20) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/handler.cc:7194
#20 0x00000000007fb52a in plugin_foreach_with_mask (thd=0x0, func=0x64d7de <exts_handlerton(THD*, plugin_ref, void*)>, type=1, state_mask=4294967287, arg=0x7f187c49bf20) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_plugin.cc:2158
#21 0x000000000064d9d6 in ha_known_exts () at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/handler.cc:7230
#22 0x00000000007ad04a in find_db_tables_and_rm_known_files (thd=0x7f1868751000, dirp=0x7f18018f3080, db=0x7f18018950b0 "test", path=0x7f187c49c920 "./test/", tables=0x7f187c49db10, found_other_files=0x7f187c49db1f) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_db.cc:1030
#23 0x00000000007ac6e0 in mysql_rm_db (thd=0x7f1868751000, db=0x7f18018950b0 "test", if_exists=false, silent=false) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_db.cc:812
#24 0x00000000007e93db in mysql_execute_command (thd=0x7f1868751000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:4354
#25 0x00000000007f01aa in mysql_parse (thd=0x7f1868751000, rawbuf=0x7f1801895010 "DROP DATABASE test", length=18, parser_state=0x7f187c49ef50) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:6955
#26 0x00000000007e1fba in dispatch_command (command=COM_QUERY, thd=0x7f1868751000, packet=0x7f1868755001 "DROP DATABASE test", packet_length=18) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:1442
#27 0x00000000007e0e28 in do_command (thd=0x7f1868751000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_parse.cc:1054
#28 0x00000000007a8dad in do_handle_one_connection (thd_arg=0x7f1862aee000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_connect.cc:1532
#29 0x00000000007a88b5 in handle_one_connection (arg=0x7f1862aee000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/sql/sql_connect.cc:1443
#30 0x0000000000dcfb11 in pfs_spawn_thread (arg=0x7f18783ff7e0) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.24-72.2/storage/perfschema/pfs.cc:1860
#31 0x00007f187bee8df5 in start_thread (arg=0x7f187c4a0700) at pthread_create.c:308
#32 0x00007f187abb21ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

The attached tarball gives the testcase as an exact match of our system, including some handy utilities

Case 1 reproducible steps

 ./{epoch}_init ; ./{epoch}_start ; sleep 2; ./{epoch}_run1 ; ./{epoch}_stop

Case 2 reproducible steps

 ./{epoch}_init ; ./{epoch}_start ; sleep 2; ./{epoch}_run2 ; ./{epoch}_run2

Revision history for this message
Ramesh Sivaraman (rameshvs02) wrote :
tags: added: set-statement
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-904

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.