POT crash on event_del()

[Valentine Gostev]

rpl.rpl_insert 'mix' w7 [ fail ]
        Test ended at 2011-05-31 15:40:13

CURRENT_TEST: rpl.rpl_insert

Server [mysqld.1 - pid: 5017, winpid: 5017, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
110601 1:40:12 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4)
mysqld: /mnt/x25m/gval/r3483/extra/libevent/event.c:771: event_del: Assertion `!(ev->ev_flags & ~(0xf000 | 0x9f))' failed.
110601 1:40:13 - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=14
max_threads=21
thread_count=10
connection_count=9
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10897 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x1ad4d850
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x414f6108 thread_stack 0x40000
/mnt/x25m/gval/r3483/sql/mysqld(my_print_stacktrace+0x32)[0x8c8316]
/mnt/x25m/gval/r3483/sql/mysqld(handle_segfault+0x342)[0x539316]
/lib64/libpthread.so.0[0x3fccc0eb10]
/lib64/libc.so.6(gsignal+0x35)[0x3fcc430265]
/lib64/libc.so.6(abort+0x110)[0x3fcc431d10]
/lib64/libc.so.6(__assert_fail+0xf6)[0x3fcc4296e6]
/mnt/x25m/gval/r3483/sql/mysqld(event_del+0x77)[0xb7e1f8]
/mnt/x25m/gval/r3483/sql/mysqld[0xb7e6f0]
/mnt/x25m/gval/r3483/sql/mysqld(event_base_loop+0x16b)[0xb7d969]
/mnt/x25m/gval/r3483/sql/mysqld(event_loop+0x1d)[0xb7d7fc]
/mnt/x25m/gval/r3483/sql/mysqld(libevent_thread_proc+0x129)[0x6afeed]
/lib64/libpthread.so.0[0x3fccc0673d]
/lib64/libc.so.6(clone+0x6d)[0x3fcc4d3f6d]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query ((nil)): Connection ID (thread ID): 519
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------
mysqltest failed but provided no output
The result from queries just before the failure was:
< snip >
#
# Bug#20821: INSERT DELAYED fails to write some rows to binlog
#
include/master-slave.inc
[connection master]
CREATE SCHEMA IF NOT EXISTS mysqlslap;
USE mysqlslap;
CREATE TABLE t1 (id INT, name VARCHAR(64));

 - saving '/mnt/x25m/gval/r3483/mysql-test/var/7/log/rpl.rpl_insert-mix/' to '/mnt/x25m/gval/r3483/mysql-test/var/log/rpl.rpl_insert-mix/'
 - found 'core.5018' (2/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace
Core generated by '/mnt/x25m/gval/r3483/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 5449]
[New Thread 5204]
[New Thread 5202]
[New Thread 5201]
[New Thread 5200]
[New Thread 5199]
[New Thread 5198]
[New Thread 5197]
[New Thread 5196]
[New Thread 5195]
[New Thread 5194]
[New Thread 5193]
[New Thread 5192]
[New Thread 5191]
[New Thread 5189]
[New Thread 5188]
[New Thread 5187]
[New Thread 5186]
[New Thread 5185]
[New Thread 5184]
[New Thread 5182]
[New Thread 5051]
[New Thread 5049]
[New Thread 5048]
[New Thread 5047]
[New Thread 5030]
[New Thread 5029]
[New Thread 5028]
[New Thread 5027]
[New Thread 5026]
[New Thread 5024]
[New Thread 5018]
Core was generated by `/mnt/x25m/gval/r3483/sql/mysqld --defaults-group-suffix=.1 --defaults-file=/mnt/'.
Program terminated with signal 6, Aborted.
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=6) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=6) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000003fcc430265 in raise () from /lib64/libc.so.6
#5 0x0000003fcc431d10 in abort () from /lib64/libc.so.6
#6 0x0000003fcc4296e6 in __assert_fail () from /lib64/libc.so.6
#7 0x0000000000b7e1f8 in event_del (ev=0x2aaab4007b10) at /mnt/x25m/gval/r3483/extra/libevent/event.c:771
#8 0x0000000000b7e6f0 in timeout_process (base=0x1a5662a0) at /mnt/x25m/gval/r3483/extra/libevent/event.c:896
#9 0x0000000000b7d969 in event_base_loop (base=0x1a5662a0, flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:506
#10 0x0000000000b7d7fc in event_loop (flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:443
#11 0x00000000006afeed in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:592
#12 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#13 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 33 (Thread 5018):
#0 0x0000003fcc4cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000539f1f in handle_connections_sockets () at /mnt/x25m/gval/r3483/sql/mysqld.cc:5198
#2 0x000000000053e936 in mysqld_main (argc=49, argv=0x1a4f4a40) at /mnt/x25m/gval/r3483/sql/mysqld.cc:4638
#3 0x0000000000533383 in main (argc=10, argv=0x7fff7e4c7458) at /mnt/x25m/gval/r3483/sql/main.cc:24

Thread 32 (Thread 5024):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab0bf30, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=0, message1=0x41f100e0, message2=0x41f100d8, type=0x41f100d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=0) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df960) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 31 (Thread 5026):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab19a90, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=1, message1=0x40e340e0, message2=0x40e340d8, type=0x40e340d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=1) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df968) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 5027):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab275f0, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=2, message1=0x429110e0, message2=0x429110d8, type=0x429110d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=2) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df970) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 5028):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab275f0, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=3, message1=0x433120e0, message2=0x433120d8, type=0x433120d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=3) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df978) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 5029):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab42950, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=4, message1=0x43d130e0, message2=0x43d130d8, type=0x43d130d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=4) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df980) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 5030):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ab42950, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=5, message1=0x447140e0, message2=0x447140d8, type=0x447140d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=5) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df988) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 5047):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1a91aa58, mp=0x1a91a9e0, abstime=0x45b16060, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1a91aa58, mutex=0x1a91a9e0, abstime=0x45b16060) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1a91a9e0, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d5ca in srv_lock_timeout_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2274
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 5048):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1a91a998, mp=0x1a91a920, abstime=0x46517070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1a91a998, mutex=0x1a91a920, abstime=0x46517070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1a91a920, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d983 in srv_error_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2456
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 5049):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1a91ab18, mp=0x1a91aaa0, abstime=0x46f18070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x1a91ab18, mutex=0x1a91aaa0, abstime=0x46f18070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x1a91aaa0, time_in_usec=5000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d226 in srv_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2124
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 5051):
#0 0x0000003fccc0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cef82 in safe_cond_wait (cond=0x1a570968, mp=0x1a5708f0, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=207) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:240
#2 0x0000000000a4cd4a in os_cond_wait (cond=0x1a570968, mutex=0x1a5708f0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:207
#3 0x0000000000a4ccc3 in os_event_wait_low (event=0x1a5708f0, reset_sig_count=0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:634
#4 0x000000000093f054 in srv_master_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:3067
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 5182):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 5184):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 20 (Thread 5185):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 19 (Thread 5186):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x12d86c8, mp=0x12d80e8, abstime=0x4799a0e0, file=0xc29610 "/mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h", line=1009) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x00000000007e60d2 in inline_mysql_cond_timedwait (that=0x12d86c8, mutex=0x12d80e8, abstime=0x4799a0e0, src_file=0xc29038 "/mnt/x25m/gval/r3483/sql/log.cc", src_line=5533) at /mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h:1009
#3 0x00000000007e6191 in MYSQL_BIN_LOG::wait_for_update_bin_log (this=0x12d80e0, thd=0x1acb1310, timeout=0x4799a0e0) at /mnt/x25m/gval/r3483/sql/log.cc:5532
#4 0x00000000005fbf8f in mysql_binlog_send (thd=0x1acb1310, log_ident=0x1acbc2f0 "master-bin.000001", pos=79586, flags=0) at /mnt/x25m/gval/r3483/sql/sql_repl.cc:874
#5 0x00000000005dd528 in dispatch_command (command=COM_BINLOG_DUMP, thd=0x1acb1310, packet=0x1acb42c1 "", packet_length=27) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:1211
#6 0x00000000005de23d in do_command (thd=0x1acb1310) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:771
#7 0x00000000006b004d in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:649
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 5187):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 17 (Thread 5188):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 16 (Thread 5189):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 15 (Thread 5191):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 14 (Thread 5192):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 13 (Thread 5193):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 5194):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 5195):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 5196):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 5197):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 5198):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 5199):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 5200):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 5201):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 4 (Thread 5202):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 5204):
#0 0x0000003fccc0e838 in do_sigwait () from /lib64/libpthread.so.0
#1 0x0000003fccc0e8dd in sigwait () from /lib64/libpthread.so.0
#2 0x0000000000538ce4 in signal_hand (arg=0x0) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2739
#3 0x0000000000b2a95d in pfs_spawn_thread (arg=0x1a5665c0) at /mnt/x25m/gval/r3483/storage/perfschema/pfs.cc:1015
#4 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 5449):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x1ad76858, mp=0x1ad767e8, abstime=0x47e2d090, file=0xb9b2e0 "/mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h", line=1009) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x00000000005bb936 in inline_mysql_cond_timedwait (that=0x1ad76858, mutex=0x1ad767e8, abstime=0x47e2d090, src_file=0xb9ae70 "/mnt/x25m/gval/r3483/sql/sql_insert.cc", src_line=2708) at /mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h:1009
#3 0x00000000005beb48 in handle_delayed_insert (arg=0x1ad73820) at /mnt/x25m/gval/r3483/sql/sql_insert.cc:2708
#4 0x0000000000b2a95d in pfs_spawn_thread (arg=0x1acc30a0) at /mnt/x25m/gval/r3483/storage/perfschema/pfs.cc:1015
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 5190):
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=6) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=6) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000003fcc430265 in raise () from /lib64/libc.so.6
#5 0x0000003fcc431d10 in abort () from /lib64/libc.so.6
#6 0x0000003fcc4296e6 in __assert_fail () from /lib64/libc.so.6
#7 0x0000000000b7e1f8 in event_del (ev=0x2aaab4007b10) at /mnt/x25m/gval/r3483/extra/libevent/event.c:771
#8 0x0000000000b7e6f0 in timeout_process (base=0x1a5662a0) at /mnt/x25m/gval/r3483/extra/libevent/event.c:896
#9 0x0000000000b7d969 in event_base_loop (base=0x1a5662a0, flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:506
#10 0x0000000000b7d7fc in event_loop (flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:443
#11 0x00000000006afeed in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:592
#12 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#13 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

[Valentine Gostev (longbow)]

(gdb) bt
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=6) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=6) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000003fcc430265 in raise () from /lib64/libc.so.6
#5 0x0000003fcc431d10 in abort () from /lib64/libc.so.6
#6 0x0000003fcc4296e6 in __assert_fail () from /lib64/libc.so.6
#7 0x0000000000b7e1f8 in event_del (ev=0x2aaab4007b10) at /mnt/x25m/gval/r3483/extra/libevent/event.c:771
#8 0x0000000000b7e6f0 in timeout_process (base=0x1a5662a0) at /mnt/x25m/gval/r3483/extra/libevent/event.c:896
#9 0x0000000000b7d969 in event_base_loop (base=0x1a5662a0, flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:506
#10 0x0000000000b7d7fc in event_loop (flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:443
#11 0x00000000006afeed in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:592
#12 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#13 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

[Valentine Gostev (longbow)]

core