Percona server crashed in plugin_var_memalloc_free

[johnrobbet]

Hi all,
I encountered a crash of Percona server, is this a new problem or known issue? Can somebody please help to take a look? Please let me know if more information is needed. Thanks in advanced.
Looks like Percona server crashed when releasing the the thread local variables related to plugins, but from the code snippet, it seems not possible crashed here.
Version 5.6.36-82.0-log
function plugin_var_memalloc_free()
code of crash:
static void plugin_var_memalloc_free(struct system_variables *vars)
{
  LIST *next, *root;
  DBUG_ENTER("plugin_var_memalloc_free");
  for (root= vars->dynamic_variables_allocs; root; root= next)
  {
    ==> next= root->next; <== crashed here
    my_free(root);
  }
  vars->dynamic_variables_allocs= NULL;
  DBUG_VOID_RETURN;
}
call stack:
(gdb) bt
#0 0x00007f0071ede621 in pthread_kill () from /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/libpthread-2.18.so
#1 0x000000000066febe in handle_fatal_signal (sig=11) at /home/mysql/opdir/output/percona-5636-trunk/sql/signal_handler.cc:236
#2 <signal handler called>
#3 plugin_var_memalloc_free (vars=0x7ee56850d648) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_plugin.cc:3179
#4 cleanup_variables (vars=0x7ee56850d648, thd=0x7ee56850d000) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_plugin.cc:2963
#5 plugin_thdvar_cleanup (thd=thd@entry=0x7ee56850d000) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_plugin.cc:2984
#6 0x00000000006c4800 in end_connection (thd=0x7ee56850d000) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_connect.cc:1324
#7 0x00000000006c4ed2 in do_handle_one_connection (thd_arg=thd_arg@entry=0x7eff1411a000) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_connect.cc:1553
#8 0x00000000006c5008 in handle_one_connection (arg=arg@entry=0x7eff1411a000) at /home/mysql/opdir/output/percona-5636-trunk/sql/sql_connect.cc:1454
#9 0x0000000000991825 in pfs_spawn_thread (arg=0x7eff86705a60) at /home/mysql/opdir/output/percona-5636-trunk/storage/perfschema/pfs.cc:1860
#10 0x00007f0071ed91c3 in start_thread () from /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/libpthread-2.18.so
#11 0x00007f0070bab12d in clone () from /home/opt/gcc-4.8.2.bpkg-r2/gcc-4.8.2.bpkg-r2/lib64/libc-2.18.so
and values of the variables:
(gdb) p *vars->dynamic_variables_allocs
$7 = {prev = 0x0, next = 0x7ee562836260, data = 0x2e302e302e373231}
(gdb) p vars->dynamic_variables_allocs->next
$9 = (st_list *) 0x7ee562836260
(gdb) p (char*)(vars->dynamic_variables_allocs->next + 1)
$10 = 0x7ee562836278 "m"
(gdb) p (char*)(vars->dynamic_variables_allocs + 1)
$11 = 0x7ee5629becf8 "off"
(gdb) p root
$12 = (LIST *) 0x7475626972747369
(gdb) p *root
Cannot access memory at address 0x7475626972747369
(gdb) p (char*)(vars->dynamic_variables_allocs->next)
$13 = 0x7f7a623ffc40 "/proc/7379/task/18180/mem"

OS:
CentOS release 4.3 (Final), Linux 2.6.32_1-18-0-0 #1 SMP Fri Jan 29 18:37:55 CST 2016 x86_64 GNU/Linux

[johnrobbet (johnrobbet)]

This crash occurs on a master database intermittently, once during several days. I googled it but looks like the problem has not been met before.

[Nickolay Ihalainen (ihanick)]

Hi,

Could you provide output for:
SHOW VARIABLES;
SHOW PLUGINS;
select * from mysql.func;

vars->dynamic_variables_allocs at this point is used for session-based variables, it's not containing "/proc/7379/task/18180/mem" string, maybe memory is already corrupted at this poit.
For safety please also check server with memtest (or similar tool).

[Launchpad Janitor (janitor)]

[Expired for Percona Server because there has been no activity for 60 days.]

[Shahriyar Rzayev (rzayev-sehriyar)]

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-3731