POT crash on free_root()

[Valentine Gostev]

rpl.rpl_mix_insert_delayed 'mix' w1 [ fail ]
        Test ended at 2011-05-31 15:53:29

CURRENT_TEST: rpl.rpl_mix_insert_delayed
/mnt/x25m/gval/r3483/client//mysqlslap: Cannot run query INSERT DELAYED INTO t1 VALUES (null, 'Dr. No'), (null, 'From Russia With Love'), (null, 'Goldfinger'), (null, 'Thunderball'), (null, 'You Only Live Twice') ERROR : Lost connection to MySQL server during query
/mnt/x25m/gval/r3483/client//mysqlslap: Cannot run query INSERT DELAYED INTO t1 VALUES (null, 'Dr. No'), (null, 'From Russia With Love'), (null, 'Goldfinger'), (null, 'Thunderball'), (null, 'You Only Live Twice') ERROR : Lost connection to MySQL server during query
mysqltest: In included file "./extra/rpl_tests/rpl_insert_delayed.test":
included from ./extra/rpl_tests/rpl_insert_delayed.test at line 24:
At line 24: query 'FLUSH TABLE t1' failed: 2006: MySQL server has gone away

The result from queries just before the failure was:
include/master-slave.inc
[connection master]
CREATE SCHEMA IF NOT EXISTS mysqlslap;
USE mysqlslap;
select @@global.binlog_format;
@@global.binlog_format
MIXED
CREATE TABLE t1 (id INT primary key auto_increment, name VARCHAR(64));

Server [mysqld.1 - pid: 25555, winpid: 25555, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
110601 1:53:28 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4)
110601 1:53:29 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=12
max_threads=21
thread_count=13
connection_count=12
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10897 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x17c7b240
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x47e07108 thread_stack 0x40000
/mnt/x25m/gval/r3483/sql/mysqld(my_print_stacktrace+0x32)[0x8c8316]
/mnt/x25m/gval/r3483/sql/mysqld(handle_segfault+0x342)[0x539316]
/lib64/libpthread.so.0[0x3fccc0eb10]
/mnt/x25m/gval/r3483/sql/mysqld(free_root+0xd5)[0x8b4882]
/mnt/x25m/gval/r3483/sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2058)[0x5ddf22]
/mnt/x25m/gval/r3483/sql/mysqld(_Z10do_commandP3THD+0x269)[0x5de23d]
/mnt/x25m/gval/r3483/sql/mysqld(libevent_thread_proc+0x289)[0x6b004d]
/lib64/libpthread.so.0[0x3fccc0673d]
/lib64/libc.so.6(clone+0x6d)[0x3fcc4d3f6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): Connection ID (thread ID): 862
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------

 - saving '/mnt/x25m/gval/r3483/mysql-test/var/1/log/rpl.rpl_mix_insert_delayed-mix/' to '/mnt/x25m/gval/r3483/mysql-test/var/log/rpl.rpl_mix_insert_delayed-mix/'
 - found 'core.25556' (9/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace
Core generated by '/mnt/x25m/gval/r3483/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 26118]
[New Thread 25692]
[New Thread 25684]
[New Thread 25683]
[New Thread 25682]
[New Thread 25681]
[New Thread 25680]
[New Thread 25679]
[New Thread 25678]
[New Thread 25670]
[New Thread 25669]
[New Thread 25667]
[New Thread 25663]
[New Thread 25661]
[New Thread 25659]
[New Thread 25658]
[New Thread 25656]
[New Thread 25655]
[New Thread 25654]
[New Thread 25653]
[New Thread 25652]
[New Thread 25624]
[New Thread 25623]
[New Thread 25622]
[New Thread 25621]
[New Thread 25605]
[New Thread 25604]
[New Thread 25603]
[New Thread 25602]
[New Thread 25601]
[New Thread 25600]
[New Thread 25556]
Core was generated by `/mnt/x25m/gval/r3483/sql/mysqld --defaults-group-suffix=.1 --defaults-file=/mnt/'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=11) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=11) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x00000000008b4882 in free_root (root=0x17c7dee0, MyFlags=1) at /mnt/x25m/gval/r3483/mysys/my_alloc.c:363
#5 0x00000000005ddf22 in dispatch_command (command=COM_QUERY, thd=0x17c7b240, packet=0x17c7e1f1 "", packet_length=155) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:1412
#6 0x00000000005de23d in do_command (thd=0x17c7b240) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:771
#7 0x00000000006b004d in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:649
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 33 (Thread 25556):
#0 0x0000003fcc4cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000539f1f in handle_connections_sockets () at /mnt/x25m/gval/r3483/sql/mysqld.cc:5198
#2 0x000000000053e936 in mysqld_main (argc=49, argv=0x17424a40) at /mnt/x25m/gval/r3483/sql/mysqld.cc:4638
#3 0x0000000000533383 in main (argc=10, argv=0x7fffc8499318) at /mnt/x25m/gval/r3483/sql/main.cc:24

Thread 32 (Thread 25600):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a3bf30, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=0, message1=0x40d9e0e0, message2=0x40d9e0d8, type=0x40d9e0d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=0) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df960) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 31 (Thread 25601):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a49a90, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=1, message1=0x421740e0, message2=0x421740d8, type=0x421740d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=1) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df968) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 25602):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a575f0, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=2, message1=0x42b750e0, message2=0x42b750d8, type=0x42b750d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=2) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df970) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 25603):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a575f0, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=3, message1=0x435760e0, message2=0x435760d8, type=0x435760d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=3) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df978) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 25604):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a72950, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=4, message1=0x43f770e0, message2=0x43f770d8, type=0x43f770d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=4) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df980) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 25605):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x17a72950, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=5, message1=0x449780e0, message2=0x449780d8, type=0x449780d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=5) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df988) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 25621):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1784aa58, mp=0x1784a9e0, abstime=0x45d7a060, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1784aa58, mutex=0x1784a9e0, abstime=0x45d7a060) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1784a9e0, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d5ca in srv_lock_timeout_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2274
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 25622):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1784a998, mp=0x1784a920, abstime=0x4677b070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1784a998, mutex=0x1784a920, abstime=0x4677b070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1784a920, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d983 in srv_error_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2456
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 25623):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1784ab18, mp=0x1784aaa0, abstime=0x4717c070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1784ab18, mutex=0x1784aaa0, abstime=0x4717c070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1784aaa0, time_in_usec=5000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d226 in srv_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2124
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 25624):
#0 0x0000003fccc0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cef82 in safe_cond_wait (cond=0x174a0968, mp=0x174a08f0, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=207) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:240
#2 0x0000000000a4cd4a in os_cond_wait (cond=0x174a0968, mutex=0x174a08f0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:207
#3 0x0000000000a4ccc3 in os_event_wait_low (event=0x174a08f0, reset_sig_count=0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:634
#4 0x000000000093f054 in srv_master_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:3067
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 25652):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x12d86c8, mp=0x12d80e8, abstime=0x47bbd0e0, file=0xc29610 "/mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h", line=1009) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x00000000007e60d2 in inline_mysql_cond_timedwait (that=0x12d86c8, mutex=0x12d80e8, abstime=0x47bbd0e0, src_file=0xc29038 "/mnt/x25m/gval/r3483/sql/log.cc", src_line=5533) at /mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h:1009
#3 0x00000000007e6191 in MYSQL_BIN_LOG::wait_for_update_bin_log (this=0x12d80e0, thd=0x17c2d3b0, timeout=0x47bbd0e0) at /mnt/x25m/gval/r3483/sql/log.cc:5532
#4 0x00000000005fbf8f in mysql_binlog_send (thd=0x17c2d3b0, log_ident=0x17bf64d0 "master-bin.000001", pos=94083, flags=0) at /mnt/x25m/gval/r3483/sql/sql_repl.cc:874
#5 0x00000000005dd528 in dispatch_command (command=COM_BINLOG_DUMP, thd=0x17c2d3b0, packet=0x17bdd481 "", packet_length=27) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:1211
#6 0x00000000005de23d in do_command (thd=0x17c2d3b0) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:771
#7 0x00000000006b004d in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:649
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 25653):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 20 (Thread 25654):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 19 (Thread 25655):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 25656):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 17 (Thread 25658):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 16 (Thread 25659):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 15 (Thread 25661):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 14 (Thread 25663):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 13 (Thread 25667):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 25669):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 25670):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 25678):
#0 0x0000003fcc473005 in _int_malloc () from /lib64/libc.so.6
#1 0x0000003fcc474cde in malloc () from /lib64/libc.so.6
#2 0x00000000008bfb23 in my_malloc (size=32, my_flags=1040) at /mnt/x25m/gval/r3483/mysys/my_malloc.c:38
#3 0x00000000008b42d6 in alloc_root (mem_root=0x17c84ea0, length=16) at /mnt/x25m/gval/r3483/mysys/my_alloc.c:217
#4 0x00000000008b4ef5 in strmake_root (root=0x17c84ea0, str=0x47f0a4f0 "mysqlslap", len=9) at /mnt/x25m/gval/r3483/mysys/my_alloc.c:421
#5 0x00000000005a715d in make_lex_string_root (mem_root=0x17c84ea0, lex_str=0x47f0aad0, str=0x47f0a4f0 "mysqlslap", length=9, allocate_lex_string=false) at /mnt/x25m/gval/r3483/sql/sql_class.cc:1430
#6 0x0000000000562f79 in parse_client_handshake_packet (mpvio=0x47f0a840, buff=0x47f0a728, pkt_len=70) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:8650
#7 0x000000000056351b in server_mpvio_read_packet (param=0x47f0a840, buf=0x47f0a728) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:8890
#8 0x000000000055ffc1 in native_password_authenticate (vio=0x47f0a840, info=0x47f0a858) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9467
#9 0x0000000000561b2c in do_auth_once (thd=0x17c82200, auth_plugin_name=0x1021b00, mpvio=0x47f0a840) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9053
#10 0x00000000005642a2 in acl_authenticate (thd=0x17c82200, connect_errors=0, com_change_user_pkt_len=0) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9189
#11 0x00000000006ad554 in check_connection (thd=0x17c82200) at /mnt/x25m/gval/r3483/sql/sql_connect.cc:523
#12 0x00000000006ad5fc in login_connection (thd=0x17c82200) at /mnt/x25m/gval/r3483/sql/sql_connect.cc:581
#13 0x00000000006b000a in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:626
#14 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#15 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 25679):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 25680):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 25681):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 25682):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 25683):
#0 0x0000003fccc0d91b in read () from /lib64/libpthread.so.0
#1 0x0000000000b71e39 in vio_read (vio=0x17c9a2d0, buf=0x17c90180 "\r\242\017", size=70) at /mnt/x25m/gval/r3483/vio/viosocket.c:44
#2 0x00000000005412bf in my_real_read (net=0x17c8d440, complen=0x4804f618) at /mnt/x25m/gval/r3483/sql/net_serv.cc:831
#3 0x0000000000541887 in my_net_read (net=0x17c8d440) at /mnt/x25m/gval/r3483/sql/net_serv.cc:1028
#4 0x00000000005634d9 in server_mpvio_read_packet (param=0x4804f840, buf=0x4804f728) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:8877
#5 0x000000000055ffc1 in native_password_authenticate (vio=0x4804f840, info=0x4804f858) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9467
#6 0x0000000000561b2c in do_auth_once (thd=0x17c8d1d0, auth_plugin_name=0x1021b00, mpvio=0x4804f840) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9053
#7 0x00000000005642a2 in acl_authenticate (thd=0x17c8d1d0, connect_errors=0, com_change_user_pkt_len=0) at /mnt/x25m/gval/r3483/sql/sql_acl.cc:9189
#8 0x00000000006ad554 in check_connection (thd=0x17c8d1d0) at /mnt/x25m/gval/r3483/sql/sql_connect.cc:523
#9 0x00000000006ad5fc in login_connection (thd=0x17c8d1d0) at /mnt/x25m/gval/r3483/sql/sql_connect.cc:581
#10 0x00000000006b000a in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:626
#11 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#12 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 4 (Thread 25684):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 25692):
#0 0x0000003fccc0e838 in do_sigwait () from /lib64/libpthread.so.0
#1 0x0000003fccc0e8dd in sigwait () from /lib64/libpthread.so.0
#2 0x0000000000538ce4 in signal_hand (arg=0x0) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2739
#3 0x0000000000b2a95d in pfs_spawn_thread (arg=0x174965c0) at /mnt/x25m/gval/r3483/storage/perfschema/pfs.cc:1015
#4 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 26118):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x17c9d628, mp=0x17c9d5b8, abstime=0x412c5090, file=0xb9b2e0 "/mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h", line=1009) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x00000000005bb936 in inline_mysql_cond_timedwait (that=0x17c9d628, mutex=0x17c9d5b8, abstime=0x412c5090, src_file=0xb9ae70 "/mnt/x25m/gval/r3483/sql/sql_insert.cc", src_line=2708) at /mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h:1009
#3 0x00000000005beb48 in handle_delayed_insert (arg=0x17c9a5f0) at /mnt/x25m/gval/r3483/sql/sql_insert.cc:2708
#4 0x0000000000b2a95d in pfs_spawn_thread (arg=0x2aaab0000e90) at /mnt/x25m/gval/r3483/storage/perfschema/pfs.cc:1015
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 25665):
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=11) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=11) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x00000000008b4882 in free_root (root=0x17c7dee0, MyFlags=1) at /mnt/x25m/gval/r3483/mysys/my_alloc.c:363
#5 0x00000000005ddf22 in dispatch_command (command=COM_QUERY, thd=0x17c7b240, packet=0x17c7e1f1 "", packet_length=155) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:1412
#6 0x00000000005de23d in do_command (thd=0x17c7b240) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:771
#7 0x00000000006b004d in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:649
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6