handle_fatal_signal (sig=11) in list_delete

[Roel Van de Paar]

Attempts to create a testcase for this issue have failed. Logging the bug with as much info as is available. It is seen a fair amount of times in our runs, but is not a 'screenfiller'.

+bt
#0 0x00007f30f5339741 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x000000000183977c in my_write_core (sig=11) at /git/percona-server_dbg/mysys/stacktrace.c:247
#2 0x0000000000e7708f in handle_fatal_signal (sig=11) at /git/percona-server_dbg/sql/signal_handler.cc:223
#3 <signal handler called>
#4 0x000000000181943b in list_delete (root=0x7f2fec01a200, element=0xd4) at /git/percona-server_dbg/mysys/list.c:46
#5 0x00007f30905f7401 in sm_thd_data_release (thd=0x7f304800e680) at /git/percona-server_dbg/plugin/scalability_metrics/scalability_metrics.c:138
#6 0x00007f30905f7bbe in sm_notify (thd=0x7f304800e680, event_class=MYSQL_AUDIT_CONNECTION_CLASS, event=0x7f30a83a0ae0) at /git/percona-server_dbg/plugin/scalability_metrics/scalability_metrics.c:398
#7 0x0000000000e78f32 in plugins_dispatch (thd=0x7f304800e680, plugin=0x7f30481569b0, arg=0x7f30a83a0a10) at /git/percona-server_dbg/sql/sql_audit.cc:1227
#8 0x0000000000e7902d in event_class_dispatch (thd=0x7f304800e680, event_class=MYSQL_AUDIT_CONNECTION_CLASS, event=0x7f30a83a0ae0) at /git/percona-server_dbg/sql/sql_audit.cc:1267
#9 0x0000000000e790d5 in event_class_dispatch_error (thd=0x7f304800e680, event_class=MYSQL_AUDIT_CONNECTION_CLASS, event_name=0x20120e0 "MYSQL_AUDIT_CONNECTION_DISCONNECT", event=0x7f30a83a0ae0) at /git/percona-server_dbg/sql/sql_audit.cc:1289
#10 0x0000000000e77b65 in mysql_audit_notify (thd=0x7f304800e680, subclass=MYSQL_AUDIT_CONNECTION_DISCONNECT, subclass_name=0x20120e0 "MYSQL_AUDIT_CONNECTION_DISCONNECT", errcode=0) at /git/percona-server_dbg/sql/sql_audit.cc:413
#11 0x00000000014b99ec in close_connection (thd=0x7f304800e680, sql_errno=0, server_shutdown=true) at /git/percona-server_dbg/sql/sql_connect.cc:1485
#12 0x0000000000e70078 in Call_close_conn::operator() (this=0x7f30a83a0cd0, closing_thd=0x7f304800e680) at /git/percona-server_dbg/sql/mysqld.cc:1008
#13 0x0000000000e74ac2 in Do_THD::operator() (this=0x7f30a83a0c40, thd=0x7f304800e680) at /git/percona-server_dbg/sql/mysqld_thd_manager.cc:46
#14 0x0000000000e7506c in std::for_each<THD**, Do_THD> (__first=0x4392980, __last=0x4392990, __f=...) at /usr/include/c++/4.8.2/bits/stl_algo.h:4417
#15 0x0000000000e7476c in Global_THD_manager::do_for_all_thd (this=0x4392970, func=0x7f30a83a0cd0) at /git/percona-server_dbg/sql/mysqld_thd_manager.cc:273
#16 0x0000000000e602fb in close_connections () at /git/percona-server_dbg/sql/mysqld.cc:1079
#17 0x0000000000e62704 in signal_hand (arg=0x0) at /git/percona-server_dbg/sql/mysqld.cc:2370
#18 0x0000000001868024 in pfs_spawn_thread (arg=0x470d920) at /git/percona-server_dbg/storage/perfschema/pfs.cc:2188
#19 0x00007f30f5334dc5 in start_thread (arg=0x7f30a83a1700) at pthread_create.c:308
#20 0x00007f30f379321d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

From error log:

stack_bottom = 0 thread_stack 0x40000
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(my_print_stacktrace+0x35)[0x18396f5]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_fatal_signal+0x411)[0xe76e98]
/lib64/libpthread.so.0(+0xf100)[0x7f30f533c100]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(list_delete+0x10)[0x181943b]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/lib/mysql/plugin/scalability_metrics.so(+0x1401)[0x7f30905f7401]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/lib/mysql/plugin/scalability_metrics.so(+0x1bbe)[0x7f30905f7bbe]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0xe78f32]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0xe7902d]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0xe790d5]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z18mysql_audit_notifyP3THD33mysql_event_connection_subclass_tPKci+0x279)[0xe77b65]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z16close_connectionP3THDjb+0x86)[0x14b99ec]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN15Call_close_connclEP3THD+0xc0)[0xe70078]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN6Do_THDclEP3THD+0x34)[0xe74ac2]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZSt8for_eachIPP3THD6Do_THDET0_T_S5_S4_+0x2c)[0xe7506c]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN18Global_THD_manager14do_for_all_thdEP11Do_THD_Impl+0x6c)[0xe7476c]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0xe602fb]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(signal_hand+0x308)[0xe62704]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
/lib64/libpthread.so.0(+0x7dc5)[0x7f30f5334dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f30f379321d]

Possible connected with https://bugs.launchpad.net/percona-server/+bug/1441139 re: plugin/scalability_metrics.so

[Roel Van de Paar (roel11)]

Bug bundle with 5 individual trials (data dir/core/error log etc.) + mysqld & libs

$ find . | grep core
./1375/data/core.7694.1000.1000.11.1464448711.mysqld
./281/data/core.2279.1000.1000.11.1464421996.mysqld
./474/data/core.18834.1000.1000.11.1464426730.mysqld
./2238/data/core.23793.1000.1000.11.1464469045.mysqld
./1112/data/core.26509.1000.1000.11.1464442293.mysqld
$ find . | grep master.err
./1375/log/master.err
./281/log/master.err
./474/log/master.err
./2238/log/master.err
./1112/log/master.err
$ find . | grep mysqld\/mysqld
./mysqld/mysqld

[Roel Van de Paar (roel11)]

From Yura; I believe this is related to the same problem with "scalability_metrics" plugin - reading from and writing to invalid(freed) memory after switching the plugin off and on. Undefined behavior, in other words.

[Roel Van de Paar (roel11)]

https://bugs.launchpad.net/percona-server/+bug/1441139

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3450