Percona Server moved to https://jira.percona.com/projects/PS

handle_fatal_signal (sig=11) in Queue<PROF_MEASUREMENT>::pop | sql/sql_profile.h:127 | handle_fatal_signal (sig=11) in my_lfind

  1. Series 5.6
  2. Bug #1441139
Bug #1441139 reported by Ramesh Sivaraman
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Percona Server moved to https://jira.percona.com/projects/PS
Won't Fix
Low
Unassigned
5.1
Invalid
Undecided
Unassigned
5.5
Won't Fix
Low
Unassigned
5.6
Won't Fix
Low
Unassigned
5.7
Won't Fix
Low
Unassigned

Bug Description

** Testcase

DROP DATABASE test;CREATE DATABASE test;USE test;
INSTALL PLUGIN scalability_metrics SONAME 'scalability_metrics.so';
set session profiling=ON;
SET GLOBAL scalability_metrics_control=ON;
SET GLOBAL scalability_metrics_control=OFF;
SET GLOBAL scalability_metrics_control=ON;

The attached tarball gives the testcase as an exact match of our system,
including some handy utilities

$ vi {epoch}_mybase # Update base path in this file (the only change
required!). For non-binary (i.e. non-tarball/non-build-binary-distribution) distributions please update SOURCE_DIR location also.
$ ./{epoch}_init # Initializes the data dir
$ ./{epoch}_start # Starts mysqld
$ ./{epoch}_cl # To check mysqld is up
$ ./{epoch}_run # Run the testcase with pquery binary(produces
output)
$ vi /dev/shm/{epoch}/error.log.out # Verify the error log
$ ./{epoch}_gdb # Brings you to a gdb prompt attached to correct
mysqld
& generated core
$ ./{epoch}_parse_core # Create {epoch}_STD.gdb and {epoch}_FULL.gdb;
standard and full var gdb stack traces
etc.

** GDB info

#0 0x00007ffce1b4a771 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000abb47a in my_write_core (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/mysys/stacktrace.c:422
#2 0x00000000007312c3 in handle_fatal_signal (sig=11) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x000000000080bf58 in Queue<PROF_MEASUREMENT>::pop (this=0x7ffcc045e180) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.h:127
#5 0x0000000000809e91 in QUERY_PROFILE::~QUERY_PROFILE (this=0x7ffcc045e150, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.cc:295
#6 0x000000000080a17d in PROFILING::~PROFILING (this=0x7ffccff72be0, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_profile.cc:349
#7 0x0000000000798db7 in THD::~THD (this=0x7ffccff71000, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_class.cc:1783
#8 0x00000000007990a2 in THD::~THD (this=0x7ffccff71000, __in_chrg=<optimized out>) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_class.cc:1840
#9 0x0000000000625aba in one_thread_per_connection_end (thd=0x7ffccff71000, block_pthread=true) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/mysqld.cc:2848
#10 0x00000000007a7dc4 in do_handle_one_connection (thd_arg=0x7ffccff71000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_connect.cc:1546
#11 0x00000000007a7815 in handle_one_connection (arg=0x7ffccff71000) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/sql/sql_connect.cc:1443
#12 0x0000000000dcc8ac in pfs_spawn_thread (arg=0x7ffcdb346240) at /mnt/workspace/percona-server-5.6-binaries-debug-yassl/label_exp/centos6-64/percona-server-5.6.23-72.1/storage/perfschema/pfs.cc:1860
#13 0x00007ffce1b45df5 in start_thread (arg=0x7ffce2103700) at pthread_create.c:308
#14 0x00007ffce080f1ad in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Revision history for this message
Ramesh Sivaraman (rameshvs02) wrote :
Laurynas Biveinis (laurynas-biveinis)
tags: added: scalability-metrics
Revision history for this message
Yura Sorokin (yura-sorokin) wrote :

The problem appears only when "libjemalloc.so" is used.
Tested with Persona Server 5.6 and libjemalloc 3.1.0, 3.2.0, 3.3.0, 3.3.1, 3.4., 3.5.0 and 3.6.0.
Without "export LD_PRELOAD=.../libjemalloc.so" the crash does not happen.

Revision history for this message
Yura Sorokin (yura-sorokin) wrote :

Thanks to Valgring the real root of the problem was found. Has nothing to do with jemalloc. It only helped to reveal the problem faster.

==3708== Thread 25:
==3708== Invalid read of size 8
==3708== at 0x1D0C3802: sm_query_finished (scalability_metrics.c:232)
==3708== by 0x1D0C3C0A: sm_notify (scalability_metrics.c:365)
==3708== by 0x778618: plugins_dispatch(THD*, st_plugin_int**, void*) (sql_audit.cc:455)
==3708== by 0x7786BF: event_class_dispatch(THD*, unsigned int, void const*) (sql_audit.cc:491)
==3708== by 0x777791: general_class_handler(THD*, unsigned int, __va_list_tag*) (sql_audit.cc:90)
==3708== by 0x7780D1: mysql_audit_notify(THD*, unsigned int, unsigned int, ...) (sql_audit.cc:217)
==3708== by 0x7EDDA2: mysql_audit_general(THD*, unsigned int, int, char const*) (sql_audit.h:196)
==3708== by 0x7F23BE: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1871)
==3708== by 0x7EFE2E: do_command(THD*) (sql_parse.cc:1054)
==3708== by 0x7B4F4D: do_handle_one_connection(THD*) (sql_connect.cc:1541)
==3708== by 0x7B49CD: handle_one_connection (sql_connect.cc:1444)
==3708== by 0xEAD6C0: pfs_spawn_thread (pfs.cc:1860)
==3708== Address 0x1808cfe0 is 0 bytes inside a block of size 32 free'd
==3708== at 0x4C2AD17: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==3708== by 0xADECDF: my_free (my_malloc.c:140)
==3708== by 0xAC069F: list_free (list.c:64)
==3708== by 0x1D0C36A1: sm_ctl_update (scalability_metrics.c:176)
==3708== by 0x80D4FA: sys_var_pluginvar::global_update(THD*, set_var*) (sql_plugin.cc:3310)
==3708== by 0x73893C: sys_var::update(THD*, set_var*) (set_var.cc:193)
==3708== by 0x739AF7: set_var::update(THD*) (set_var.cc:679)
==3708== by 0x739675: sql_set_variables(THD*, List<set_var_base>*) (set_var.cc:579)
==3708== by 0x7F7CA1: mysql_execute_command(THD*) (sql_parse.cc:4177)
==3708== by 0x7FF15C: mysql_parse(THD*, char*, unsigned int, Parser_state*) (sql_parse.cc:6972)
==3708== by 0x7F0FB8: dispatch_command(enum_server_command, THD*, char*, unsigned int) (sql_parse.cc:1442)
==3708== by 0x7EFE2E: do_command(THD*) (sql_parse.cc:1054)

After executing "SET GLOBAL scalability_metrics_control=OFF" we end up calling "sm_ctl_update()" ("scalability_metrics.c").
This function clears "thd_list_root" ("list_free(thd_list_root, TRUE);") and, because of the "TRUE" parameter, frees memory for each element in the list.
However, pointers to the deleted elements still exist in thread/connection descriptors (THD structures) as they were set in "sm_thd_data_get()" ("THDVAR(thd, thd_data)= (ulonglong) (intptr) thd_data;") when these connections were created.

Therefore, next time when scalability metrics are reenabled ("SET GLOBAL scalability_metrics_control=ON") and "sm_query_started()"/ "sm_query_finished()" are called, they will be using a pointer to freed data to store stats.

Revision history for this message
Roel Van de Paar (roel11) wrote :

See also bug 1544398

Revision history for this message
Roel Van de Paar (roel11) wrote :

I am also seeing various other sporadic bugs when

SET GLOBAL scalability_metrics_control = OFF;
SET GLOBAL scalability_metrics_control = ON;

is used, perhaps in combination with other SQL.

An example stack;

+bt
#0 0x00007f007410e741 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000abf206 in my_write_core (sig=11) at /git/PS-5.6.30_dbg/mysys/stacktrace.c:422
#2 0x0000000000735c63 in handle_fatal_signal (sig=11) at /git/PS-5.6.30_dbg/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x0000000000e6acd6 in my_lfind (head=0x7efffc031ce8, cs=0x1501e20 <my_charset_bin>, hashnr=66676499, key=0x7f0043665170 " \353Z\203\251l\306\021\223\\\332A\214\202\345\062test", keylen=212, cursor=0x7f00436650d0, pins=0x7efffc01e440) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:95
#5 0x0000000000e6b154 in my_lsearch (head=0x7efffc031ce8, cs=0x1501e20 <my_charset_bin>, hashnr=66676499, key=0x7f0043665170 " \353Z\203\251l\306\021\223\\\332A\214\202\345\062test", keylen=212, pins=0x7efffc01e440) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:267
#6 0x0000000000e6b82d in lf_hash_search (hash=0x17ee180 <digest_hash>, pins=0x7efffc01e440, key=0x7f0043665170, keylen=212) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:462
#7 0x0000000000adf05e in find_or_create_digest (thread=0x7f005d732e40, digest_storage=0x2106e50, schema_name=0x2106f38 "test", schema_name_length=4) at /git/PS-5.6.30_dbg/storage/perfschema/pfs_digest.cc:223
#8 0x0000000000b14954 in end_statement_v1 (locker=0x2106e90, stmt_da=0x2108678) at /git/PS-5.6.30_dbg/storage/perfschema/pfs.cc:4815
#9 0x00000000007e2ad8 in inline_mysql_end_statement (locker=0x2106e90, stmt_da=0x2108678) at /git/PS-5.6.30_dbg/include/mysql/psi/mysql_statement.h:215
#10 0x00000000007e788b in dispatch_command (command=COM_QUERY, thd=0x2105200, packet=0x2227be1 "", packet_length=73) at /git/PS-5.6.30_dbg/sql/sql_parse.cc:1893
#11 0x00000000007e519f in do_command (thd=0x2105200) at /git/PS-5.6.30_dbg/sql/sql_parse.cc:1053
#12 0x00000000007ac701 in do_handle_one_connection (thd_arg=0x2105200) at /git/PS-5.6.30_dbg/sql/sql_connect.cc:1541
#13 0x00000000007ac181 in handle_one_connection (arg=0x2105200) at /git/PS-5.6.30_dbg/sql/sql_connect.cc:1444
#14 0x0000000000b0e4c9 in pfs_spawn_thread (arg=0x21021c0) at /git/PS-5.6.30_dbg/storage/perfschema/pfs.cc:1860
#15 0x00007f0074109dc5 in start_thread (arg=0x7f0043666700) at pthread_create.c:308
#16 0x00007f007256821d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Revision history for this message
Roel Van de Paar (roel11) wrote :
Download full text (3.8 KiB)

Another one:

+bt
#0 0x00007f5b5a3a3741 in __pthread_kill (threadid=<optimized out>, signo=11) at ../nptl/sysdeps/unix/sysv/linux/pthread_kill.c:61
#1 0x0000000000abf206 in my_write_core (sig=11) at /git/PS-5.6.30_dbg/mysys/stacktrace.c:422
#2 0x0000000000735c63 in handle_fatal_signal (sig=11) at /git/PS-5.6.30_dbg/sql/signal_handler.cc:236
#3 <signal handler called>
#4 0x0000000000e6acd6 in my_lfind (head=0x7f5a84046528, cs=0x1501e20 <my_charset_bin>, hashnr=620756992, key=0x108ae5a "", keylen=0, cursor=0x7f5b5a9a6ed0, pins=0x7f5a84093ec0) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:95
#5 0x0000000000e6aecd in linsert (head=0x7f5a84046528, cs=0x1501e20 <my_charset_bin>, node=0x7f5a840ed4a0, pins=0x7f5a84093ec0, flags=1) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:161
#6 0x0000000000e6b952 in initialize_bucket (hash=0x17ee180 <digest_hash>, node=0x7f5a84046928, bucket=164, pins=0x7f5a84093ec0) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:490
#7 0x0000000000e6b8f0 in initialize_bucket (hash=0x17ee180 <digest_hash>, node=0x7f5a8409c728, bucket=1188, pins=0x7f5a84093ec0) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:485
#8 0x0000000000e6b8f0 in initialize_bucket (hash=0x17ee180 <digest_hash>, node=0x7f5a840c2928, bucket=3236, pins=0x7f5a84093ec0) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:485
#9 0x0000000000e6b8f0 in initialize_bucket (hash=0x17ee180 <digest_hash>, node=0x7f5a840c1f28, bucket=7332, pins=0x7f5a84093ec0) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:485
#10 0x0000000000e6b7e3 in lf_hash_search (hash=0x17ee180 <digest_hash>, pins=0x7f5a84093ec0, key=0x7f5b5a9a70f0, keylen=212) at /git/PS-5.6.30_dbg/mysys/lf_hash.c:460
#11 0x0000000000adf05e in find_or_create_digest (thread=0x7f5b3f61c200, digest_storage=0x7f5b56ab8450, schema_name=0x7f5b56ab8538 "test", schema_name_length=4) at /git/PS-5.6.30_dbg/storage/perfschema/pfs_digest.cc:223
#12 0x0000000000b14954 in end_statement_v1 (locker=0x7f5b56ab8490, stmt_da=0x7f5b56ab9c78) at /git/PS-5.6.30_dbg/storage/perfschema/pfs.cc:4815
#13 0x00000000007e2ad8 in inline_mysql_end_statement (locker=0x7f5b56ab8490, stmt_da=0x7f5b56ab9c78) at /git/PS-5.6.30_dbg/include/mysql/psi/mysql_statement.h:215
#14 0x00000000007e788b in dispatch_command (command=COM_QUERY, thd=0x7f5b56ab6800, packet=0x7f5aa3b2e9c1 "", packet_length=37) at /git/PS-5.6.30_dbg/sql/sql_parse.cc:1893
#15 0x00000000007e519f in do_command (thd=0x7f5b56ab6800) at /git/PS-5.6.30_dbg/sql/sql_parse.cc:1053
#16 0x00000000007ac701 in do_handle_one_connection (thd_arg=0x7f5b56ab6800) at /git/PS-5.6.30_dbg/sql/sql_connect.cc:1541
#17 0x00000000007ac181 in handle_one_connection (arg=0x7f5b56ab6800) at /git/PS-5.6.30_dbg/sql/sql_connect.cc:1444
#18 0x0000000000b0e4c9 in pfs_spawn_thread (arg=0x7f5b3cd789a0) at /git/PS-5.6.30_dbg/storage/perfschema/pfs.cc:1860
#19 0x00007f5b5a39edc5 in start_thread (arg=0x7f5b5a9a8700) at pthread_create.c:308
#20 0x00007f5b587fd21d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113

Testcase that seems to be (sporadically?) reproducible on 5.6.30;

MYEXTRA=" --no-defaults --log-output=none --sql_mode=ONLY_FULL_GROUP_BY --plugin-load-add=audit_log=audit_log.so --plugin-load-add=tokudb=ha_tokudb.so --init-file=/home/roel/percona-qa...

Read more...

Revision history for this message
Roel Van de Paar (roel11) wrote :

Please note you may have to run the testcase few times in a row in the mysql cli.

Revision history for this message
Roel Van de Paar (roel11) wrote :

Though not a real blocker, would be good to see this one fixed

summary: handle_fatal_signal (sig=11) in Queue<PROF_MEASUREMENT>::pop |
- sql/sql_profile.h:127
+ sql/sql_profile.h:127 | handle_fatal_signal (sig=11) in my_lfind
Roel Van de Paar (roel11)
tags: added: qafix
Revision history for this message
Roel Van de Paar (roel11) wrote :

This testcase:

INSTALL PLUGIN scalability_metrics SONAME 'scalability_metrics.so';
DROP DATABASE transforms;CREATE DATABASE transforms;DROP DATABASE test;CREATE DATABASE test;USE test;
CREATE TABLE t1(c1 SMALLINT ZEROFILL NULL);
CREATE TABLE t5(c1 REAL NOT NULL);
SET GLOBAL scalability_metrics_control = OFF;
SET GLOBAL scalability_metrics_control = ON;
CREATE TABLE t577 (c1 VARCHAR(10));
Connection ID (thread ID): 6; ;

Can crash the server. It can also produce a GLIBC crash. But it is very sporadic, and hard to trigger. Here are the mysqld options used;

--log-output=none --sql_mode=ONLY_FULL_GROUP_BY --plugin-load-add=audit_log=audit_log.so --plugin-load-add=tokudb=ha_tokudb.so --init-file=/home/roel/percona-qa/plugins.sql --tokudb-check-jemalloc=0 --performance-schema-max-rwlock-instances=0 --innodb-max-dirty-pages-pct-lwm=1 --log-slow-filter=filesort_on_disk --innodb-ft-sort-pll-degree=-1125899906842624 --tokudb_fanout=32 --tokudb-read-status-frequency=0 --tokudb-fs-reserve-percent=24 --tokudb_fanout=64 --tokudb_block_size=56732 --tokudb-cleaner-iterations=2"

Though likely many (especially the TokuDB ones) are not required. Sorry to not be more specific then this, it was not straightforward to get this far.

Revision history for this message
Roel Van de Paar (roel11) wrote :
Download full text (32.0 KiB)

One of the GLIBC crashes. Even though it shows as a TokuDB issue, it could well be related to the OFF/ON scalability_metrics_control

*** Error in `/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld': malloc(): smallbin double linked list corrupted: 0x00007f0ee8042e20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7b184)[0x7f0f5f3f8184]
/lib64/libc.so.6(+0x7e877)[0x7f0f5f3fb877]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f0f5f3fc87c]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/lib/mysql/plugin/ha_tokudb.so(_Z9os_mallocm+0x18)[0x7f0f431f892a]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/lib/mysql/plugin/ha_tokudb.so(_Z12toku_xmallocm+0x4e)[0x7f0f431f5818]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/lib/mysql/plugin/ha_tokudb.so(toku_dbt_array_init+0x3b)[0x7f0f431d1b95]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/lib/mysql/plugin/ha_tokudb.so(_ZN9ha_tokudbC1EP10handlertonP11TABLE_SHARE+0x286)[0x7f0f430b8fbe]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/lib/mysql/plugin/ha_tokudb.so(+0x9d3fd)[0x7f0f430e83fd]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z15get_new_handlerP11TABLE_SHAREP11st_mem_rootP10handlerton+0xb5)[0x64401d]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld[0x859da1]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z26mysql_create_table_no_lockP3THDPKcS2_P24st_ha_create_informationP10Alter_infojPb+0x170)[0x85b227]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z18mysql_create_tableP3THDP10TABLE_LISTP24st_ha_create_informationP10Alter_info+0x135)[0x85b35e]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z21mysql_execute_commandP3THD+0x21c9)[0x7eb0af]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x581)[0x7f445b]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0xd09)[0x7e631d]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z10do_commandP3THD+0x3bb)[0x7e519f]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(_Z24do_handle_one_connectionP3THD+0x24c)[0x7ac701]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(handle_one_connection+0x33)[0x7ac181]
/sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x159)[0xb0e4c9]
/lib64/libpthread.so.0(+0x7dc5)[0x7f0f61014dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f0f5f47321d]
======= Memory map: ========
00400000-01202000 r-xp 00000000 08:00 51916505 /sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld
01401000-014ac000 r--p 00e01000 08:00 51916505 /sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld
014ac000-017ac000 rw-p 00eac000 08:00 51916505 /sda/PS180516-percona-server-5.6.30-76.3-linux-x86_64-debug/bin/mysqld
017ac000-01814000 rw-p 00000000 00:00 0
02d57000-030cd000 rw-p 00000000 00:00 0 [heap]
7f0...

Revision history for this message
Roel Van de Paar (roel11) wrote :

Demangled and cleaned stack

======= Backtrace: =========
/lib64/libc.so.6(+0x7b184)[0x7f0f5f3f8184]
/lib64/libc.so.6(+0x7e877)[0x7f0f5f3fb877]
/lib64/libc.so.6(__libc_malloc+0x4c)[0x7f0f5f3fc87c]
plugin/ha_tokudb.so(os_malloc(unsigned long)+0x18)[0x7f0f431f892a]
plugin/ha_tokudb.so(toku_xmalloc(unsigned long)+0x4e)[0x7f0f431f5818]
plugin/ha_tokudb.so(toku_dbt_array_init+0x3b)[0x7f0f431d1b95]
plugin/ha_tokudb.so(ha_tokudb::ha_tokudb(handlerton*, TABLE_SHARE*)+0x286)[0x7f0f430b8fbe]
plugin/ha_tokudb.so(+0x9d3fd)[0x7f0f430e83fd]
mysqld(get_new_handler(TABLE_SHARE*, st_mem_root*, handlerton*)+0xb5)[0x64401d]
mysqld[0x859da1]
mysqld(mysql_create_table_no_lock(THD*, char const*, char const*, st_ha_create_information*, Alter_info*, unsigned int, bool*)+0x170)[0x85b227]
mysqld(mysql_create_table(THD*, TABLE_LIST*, st_ha_create_information*, Alter_info*)+0x135)[0x85b35e]
mysqld(mysql_execute_command(THD*)+0x21c9)[0x7eb0af]
mysqld(mysql_parse(THD*, char*, unsigned int, Parser_state*)+0x581)[0x7f445b]
mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0xd09)[0x7e631d]
mysqld(do_command(THD*)+0x3bb)[0x7e519f]
mysqld(do_handle_one_connection(THD*)+0x24c)[0x7ac701]
mysqld(handle_one_connection+0x33)[0x7ac181]
mysqld(pfs_spawn_thread+0x159)[0xb0e4c9]
/lib64/libpthread.so.0(+0x7dc5)[0x7f0f61014dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f0f5f47321d]

Revision history for this message
Roel Van de Paar (roel11) wrote :

Ignore the testcase in #9. Got a good version which always works in terms of producing a mysqld crash. Uploading below. The GLIBC is not produced, but perhaps once in a while it could be (reduction was done on the GLIBC crash occurring, not mysqld crashing). It is highly sporadic, though regularly seen during overall QA trials. Then again; there are different GLIBC crashes. It is unclear if they are all related to the ON/OFF of scalability_metrics_control.

Revision history for this message
Roel Van de Paar (roel11) wrote :
Revision history for this message
Roel Van de Paar (roel11) wrote :

Note that percona-qa is necessary for this testcase because the testcase has this in start:

1464402108]$ cat 1464402108_start | grep plugins.sql
...--init-file=/home/roel/percona-qa/plugins.sql...

Otherwise, use is always, see 1464402108_how_to_use.txt in tarball.

Revision history for this message
Roel Van de Paar (roel11) wrote :

s/is/as/

Revision history for this message
Roel Van de Paar (roel11) wrote :
Download full text (6.4 KiB)

Another set of GLIBC crashes seen in combination with ON/OFF scalability_metrics_control

*** Error in `/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': free(): invalid next size (fast): 0x00007f5f94026b30 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7cfe1)[0x7f6042b6afe1]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1833f84]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(my_free+0x90)[0x1833d02]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x153caeb]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z18plugin_unlock_listP3THDPPP13st_plugin_intm+0xce)[0x153cde0]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN14Trans_delegate15before_rollbackEP3THDb+0x269)[0x14390cd]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z15ha_rollback_lowP3THDb+0x85)[0xf04a9e]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_ZN11TC_LOG_MMAP8rollbackEP3THDb+0x27)[0x1613b55]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z19trans_rollback_stmtP3THD+0x125)[0x1616885]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z21mysql_execute_commandP3THDb+0x7293)[0x1515922]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x685)[0x151707a]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0xbf8)[0x150c035]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z10do_commandP3THD+0x520)[0x150af4f]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0x2a0)[0x16447eb]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
/lib64/libpthread.so.0(+0x7dc5)[0x7f6044785dc5]
/lib64/libc.so.6(clone+0x6d)[0x7f6042be421d]

/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': corrupted double-linked list: 0x00007ffdb4026b20 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7b184)[0x7ffe642aa184]
/lib64/libc.so.6(+0x7d070)[0x7ffe642ac070]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1833f84]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(my_free+0x90)[0x1833d02]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld[0x1541fa4]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z21plugin_thdvar_cleanupP3THDb+0x18c)[0x1542154]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(_Z14end_connectionP3THD+0x49)[0x14b933e]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(handle_connection+0x2c2)[0x164480d]
/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld(pfs_spawn_thread+0x170)[0x1868024]
/lib64/libpthread.so.0(+0x7dc5)[0x7ffe65ec6dc5]
/lib64/libc.so.6(clone+0x6d)[0x7ffe6432521d]

*** Error in `/sda/PS280516-percona-server-5.7.12-5-linux-x86_64-debug/bin/mysqld': corrupted double-linked list: 0x00007fb90c026b50 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7bea2)[0x7fb9bd9e4ea2]
/lib64/libc.so.6(+0x7de...

Read more...

Revision history for this message
Roel Van de Paar (roel11) wrote :
Revision history for this message
Roel Van de Paar (roel11) wrote :

Also see bug 1587738

Laurynas Biveinis (laurynas-biveinis)
Changed in percona-server:
assignee: Laurynas Biveinis (laurynas-biveinis) → nobody
importance: High → Low
status: Triaged → Won't Fix
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PS-2112

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.