sql/mysqld.cc:873: void remove_global_thread(THD*): Assertion `1 == num_erased' failed | abort in remove_global_thread (sig=6)

[Roel Van de Paar]

Thread 28 (LWP 15107):
+bt
#0 0x000000340cc0c69c in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000006f08cd in handle_fatal_signal (sig=6) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/signal_handler.cc:251
#2 <signal handler called>
#3 0x000000340c4328a5 in raise () from /lib64/libc.so.6
#4 0x000000340c434085 in abort () from /lib64/libc.so.6
#5 0x000000340c42ba1e in __assert_fail_base () from /lib64/libc.so.6
#6 0x000000340c42bae0 in __assert_fail () from /lib64/libc.so.6
#7 0x00000000006303de in remove_global_thread (thd=0x246efbf0) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/mysqld.cc:873
#8 0x0000000000767eba in Delayed_insert::~Delayed_insert (this=0x246efbd0, __in_chrg=<optimized out>) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_insert.cc:2263
#9 0x0000000000767f81 in Delayed_insert::~Delayed_insert (this=0x246efbd0, __in_chrg=<optimized out>) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_insert.cc:2268
#10 0x00000000007636b6 in delayed_get_table (thd=0x22721370, grl_protection_request=<optimized out>, table_list=0x217af9a0) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_insert.cc:2437
#11 0x0000000000763d1b in open_and_lock_for_insert_delayed (thd=0x22721370, table_list=0x217af9a0) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_insert.cc:564
#12 0x00000000007654c5 in mysql_insert (thd=0x22721370, table_list=0x217af9a0, fields=..., values_list=..., update_fields=..., update_values=..., duplic=DUP_ERROR, ignore=false) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_insert.cc:732
#13 0x000000000077d2f7 in mysql_execute_command (thd=0x22721370) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:3647
#14 0x000000000077fc78 in mysql_parse (thd=0x22721370, rawbuf=0x22721378 "@$B\001", length=<optimized out>, parser_state=<optimized out>) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:6477
#15 0x00000000007807bd in dispatch_command (command=<optimized out>, thd=0x22721370, packet=0x23f5b9e0 "p\023r\"", packet_length=<optimized out>) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:1355
#16 0x00000000007814f5 in do_command (thd=0x22721370) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/sql_parse.cc:1049
#17 0x000000000081f2bf in threadpool_process_request (thd=0x22721370) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/threadpool_common.cc:312
#18 0x0000000000821c10 in handle_event (connection=0x131ce7b0) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/threadpool_unix.cc:1466
#19 0x0000000000821cd8 in worker_main (param=0x1576800 <all_groups+2560>) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/threadpool_unix.cc:1519
#20 0x00000000009ea66a in pfs_spawn_thread (arg=0x19548720) at /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/storage/perfschema/pfs.cc:1858
#21 0x000000340cc07851 in start_thread () from /lib64/libpthread.so.0
#22 0x000000340c4e890d in clone () from /lib64/libc.so.6

mysqld: /bzr/5.6-QA_val/Percona-Server-5.6.12-rc60.4/sql/mysqld.cc:873: void remove_global_thread(THD*): Assertion `1 == num_erased' failed.
10:40:35 UTC - mysqld got signal 6 ;
[...]
2013-08-13T20:40:35 [15089] Query: DELETE FROM `table20_int_autoinc` WHERE `col_char_10` = 'j' ORDER BY `col_bigint`,`col_bigint_default_null`,`col_bigint_key`,`col_bigint_key_default_null`,`col_bigint_not_null`,`col_bigint_not_null_key`,`col_char_10`,`col_char_10_default_null`,`col_char_10_key`,`col_char_10_key_default_null`,`col_char_10_not_null`,`col_char_10_not_null_key`,`col_char_255`,`col_char_255_default_null`,`col_char_255_key`,`col_char_255_key_default_null`,`col_char_255_not_null`,`col_char_255_not_null_key`,`col_enum`,`col_enum_default_null`,`col_enum_key`,`col_enum_key_default_null`,`col_enum_not_null`,`col_enum_not_null_key`,`col_int`,`col_int_default_null`,`col_int_key`,`col_int_key_default_null`,`col_int_not_null`,`col_int_not_null_key`,`pk` LIMIT 1 /*Generated by THREAD_ID 48*/

[Laurynas Biveinis (laurynas-biveinis)]

Threadpool run, Sergei, can you take a look?

[Sergei Glushchenko (sergei.glushchenko)]

Upstream bug. What happened here is that server was not able to
create new thread to handle delayed insert (out of resources?).
Thread was not added to the global list of threads (it happens
in thread worker routine). On resources cleanup (::~Delayed_insert)
remove_global_thread was called which led to assertion.

[Laurynas Biveinis (laurynas-biveinis)]

Sergei - thanks. Can you report it to upstream and link here? It looks like code analysis should be enough for that, no need for a testcase.

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-713