mysqld crashes on min_heap_erase while using POT

[Valentine Gostev]

Environment - r900
source code configured with:
cmake . -DWITH_DEBUG=ON -DWITH_VALGRIND=ON -DCMAKE_INSTALL_PREFIX=/mnt/x25m/gval/bin-r3483 -DWITH_LIBEVENT=bundled

rpl.rpl_insert 'row' w7 [ fail ]
        Test ended at 2011-05-31 13:32:07

CURRENT_TEST: rpl.rpl_insert

Server [mysqld.1 - pid: 5046, winpid: 5046, exit: 256] failed during test run
Server log from this test:
----------SERVER LOG START-----------
110531 23:32:02 [Note] Start binlog_dump to slave_server(2), pos(master-bin.000001, 4)
110531 23:32:06 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.

key_buffer_size=1048576
read_buffer_size=131072
max_used_connections=17
max_threads=21
thread_count=2
connection_count=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10897 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = (nil) thread_stack 0x40000
/mnt/x25m/gval/r3483/sql/mysqld(my_print_stacktrace+0x32)[0x8c8316]
/mnt/x25m/gval/r3483/sql/mysqld(handle_segfault+0x342)[0x539316]
/lib64/libpthread.so.0[0x3fccc0eb10]
/mnt/x25m/gval/r3483/sql/mysqld[0xb7e95f]
/mnt/x25m/gval/r3483/sql/mysqld[0xb7e8ae]
/mnt/x25m/gval/r3483/sql/mysqld(event_del+0xbe)[0xb7e23f]
/mnt/x25m/gval/r3483/sql/mysqld(libevent_thread_proc+0x196)[0x6aff5a]
/lib64/libpthread.so.0[0x3fccc0673d]
/lib64/libc.so.6(clone+0x6d)[0x3fcc4d3f6d]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
Writing a core file
----------SERVER LOG END-------------
mysqltest failed but provided no output
The result from queries just before the failure was:
< snip >
#
# Bug#20821: INSERT DELAYED fails to write some rows to binlog
#
include/master-slave.inc
[connection master]
CREATE SCHEMA IF NOT EXISTS mysqlslap;
USE mysqlslap;
CREATE TABLE t1 (id INT, name VARCHAR(64));
SELECT COUNT(*) FROM mysqlslap.t1;
COUNT(*)
5000
SELECT COUNT(*) FROM mysqlslap.t1;
COUNT(*)
5000
#
# Cleanup
#
USE test;
DROP SCHEMA mysqlslap;
include/rpl_end.inc

 - saving '/mnt/x25m/gval/r3483/mysql-test/var/7/log/rpl.rpl_insert-row/' to '/mnt/x25m/gval/r3483/mysql-test/var/log/rpl.rpl_insert-row/'
 - found 'core.5047' (0/5)

Trying 'dbx' to get a backtrace

Trying 'gdb' to get a backtrace
Core generated by '/mnt/x25m/gval/r3483/sql/mysqld'
Output from gdb follows. The first stack trace is from the failing thread.
The following stack traces are from all threads (so the failing one is
duplicated).
--------------------------
[New Thread 5186]
[New Thread 5184]
[New Thread 5182]
[New Thread 5181]
[New Thread 5180]
[New Thread 5179]
[New Thread 5178]
[New Thread 5176]
[New Thread 5175]
[New Thread 5174]
[New Thread 5173]
[New Thread 5172]
[New Thread 5171]
[New Thread 5170]
[New Thread 5169]
[New Thread 5168]
[New Thread 5167]
[New Thread 5166]
[New Thread 5165]
[New Thread 5164]
[New Thread 5124]
[New Thread 5123]
[New Thread 5122]
[New Thread 5121]
[New Thread 5063]
[New Thread 5062]
[New Thread 5061]
[New Thread 5060]
[New Thread 5059]
[New Thread 5058]
[New Thread 5047]
Core was generated by `/mnt/x25m/gval/r3483/sql/mysqld --defaults-group-suffix=.1 --defaults-file=/mnt/'.
Program terminated with signal 11, Segmentation fault.
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=11) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=11) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000000000b7e95f in min_heap_erase (s=0x15e74a8, e=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/min_heap.h:88
#5 0x0000000000b7e8ae in event_queue_remove (base=0x15e72a0, ev=0x2aaab0001070, queue=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:925
#6 0x0000000000b7e23f in event_del (ev=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/event.c:780
#7 0x00000000006aff5a in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:606
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 32 (Thread 5047):
#0 0x0000003fcc4cb186 in poll () from /lib64/libc.so.6
#1 0x0000000000539f1f in handle_connections_sockets () at /mnt/x25m/gval/r3483/sql/mysqld.cc:5198
#2 0x000000000053e936 in mysqld_main (argc=49, argv=0x1575a40) at /mnt/x25m/gval/r3483/sql/mysqld.cc:4638
#3 0x0000000000533383 in main (argc=10, argv=0x7fffcf097988) at /mnt/x25m/gval/r3483/sql/main.cc:24

Thread 31 (Thread 5058):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1b8cf30, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=0, message1=0x4201f0e0, message2=0x4201f0d8, type=0x4201f0d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=0) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df960) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 30 (Thread 5059):
#0 0x0000003fccc0e047 in fsync () from /lib64/libpthread.so.0
#1 0x0000000000a472b4 in os_file_fsync (file=9) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:2058
#2 0x0000000000a47385 in os_file_flush_func (file=9) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:2156
#3 0x00000000009efeac in pfs_os_file_flush_func (file=9, src_file=0xc7d7e0 "/mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c", src_line=4647) at /mnt/x25m/gval/r3483/storage/innobase/include/os0file.ic:381
#4 0x00000000009f3076 in fil_flush (space_id=4294967280) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4647
#5 0x0000000000a302df in log_io_complete (group=0x1c4f048) at /mnt/x25m/gval/r3483/storage/innobase/log/log0log.c:1105
#6 0x00000000009f2e8a in fil_aio_wait (segment=1) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4563
#7 0x000000000093ffa5 in io_handler_thread (arg=0x12df968) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 29 (Thread 5060):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ba85f0, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=2, message1=0x42a200e0, message2=0x42a200d8, type=0x42a200d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=2) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df970) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 28 (Thread 5061):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1ba85f0, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=3, message1=0x434210e0, message2=0x434210d8, type=0x434210d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=3) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df978) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 27 (Thread 5062):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1bc3950, segment=0, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=4, message1=0x43e220e0, message2=0x43e220d8, type=0x43e220d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=4) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df980) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 26 (Thread 5063):
#0 0x0000003c462005b4 in ?? () from /usr/lib64/libaio.so.1
#1 0x0000000000a49e1c in os_aio_linux_collect (array=0x1bc3950, segment=1, seg_size=256) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4420
#2 0x0000000000a4a157 in os_aio_linux_handle (global_seg=5, message1=0x448230e0, message2=0x448230d8, type=0x448230d0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0file.c:4553
#3 0x00000000009f2d63 in fil_aio_wait (segment=5) at /mnt/x25m/gval/r3483/storage/innobase/fil/fil0fil.c:4527
#4 0x000000000093ffa5 in io_handler_thread (arg=0x12df988) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0start.c:482
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 25 (Thread 5121):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x199ba58, mp=0x199b9e0, abstime=0x45c25060, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x199ba58, mutex=0x199b9e0, abstime=0x45c25060) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x199b9e0, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d5ca in srv_lock_timeout_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2274
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 24 (Thread 5122):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x199b998, mp=0x199b920, abstime=0x46626070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x199b998, mutex=0x199b920, abstime=0x46626070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x199b920, time_in_usec=1000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d983 in srv_error_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2456
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 23 (Thread 5123):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x199bb18, mp=0x199baa0, abstime=0x47027070, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=171) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x0000000000a4cf81 in os_cond_wait_timed (cond=0x199bb18, mutex=0x199baa0, abstime=0x47027070) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:171
#3 0x0000000000a4cf0e in os_event_wait_time_low (event=0x199baa0, time_in_usec=5000000, reset_sig_count=1) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:743
#4 0x000000000093d226 in srv_monitor_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2124
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 22 (Thread 5124):
#0 0x0000003fccc0aee9 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cef82 in safe_cond_wait (cond=0x1c48118, mp=0x1c480a0, file=0xc93e90 "/mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c", line=207) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:240
#2 0x0000000000a4cd4a in os_cond_wait (cond=0x1c48118, mutex=0x1c480a0) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:207
#3 0x0000000000a4ccc3 in os_event_wait_low (event=0x1c480a0, reset_sig_count=3) at /mnt/x25m/gval/r3483/storage/innobase/os/os0sync.c:634
#4 0x00000000009449e2 in sync_array_wait_event (arr=0x1576210, index=0) at /mnt/x25m/gval/r3483/storage/innobase/sync/sync0arr.c:458
#5 0x00000000009461f8 in rw_lock_s_lock_spin (lock=0x1c47d08, pass=0, file_name=0xc8db28 "/mnt/x25m/gval/r3483/storage/innobase/log/log0log.c", line=2049) at /mnt/x25m/gval/r3483/storage/innobase/sync/sync0rw.c:456
#6 0x0000000000a32409 in rw_lock_s_lock_func (lock=0x1c47d08, pass=0, file_name=0xc8db28 "/mnt/x25m/gval/r3483/storage/innobase/log/log0log.c", line=2049) at /mnt/x25m/gval/r3483/storage/innobase/include/sync0rw.ic:418
#7 0x0000000000a3239d in pfs_rw_lock_s_lock_func (lock=0x1c47d08, pass=0, file_name=0xc8db28 "/mnt/x25m/gval/r3483/storage/innobase/log/log0log.c", line=2049) at /mnt/x25m/gval/r3483/storage/innobase/include/sync0rw.ic:782
#8 0x0000000000a322b2 in log_checkpoint (sync=1, write_always=0) at /mnt/x25m/gval/r3483/storage/innobase/log/log0log.c:2049
#9 0x000000000093ef21 in srv_master_thread (arg=0x0) at /mnt/x25m/gval/r3483/storage/innobase/srv/srv0srv.c:2994
#10 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#11 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 21 (Thread 5164):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 20 (Thread 5165):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 19 (Thread 5166):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 18 (Thread 5167):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 17 (Thread 5168):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 16 (Thread 5169):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 15 (Thread 5170):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 14 (Thread 5171):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 13 (Thread 5172):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 12 (Thread 5173):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 11 (Thread 5174):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 10 (Thread 5175):
#0 0x0000003fce0044eb in clock_gettime () from /lib64/librt.so.1
#1 0x0000000000b7cd52 in gettime (tp=0x47d34020) at /mnt/x25m/gval/r3483/extra/libevent/event.c:146
#2 0x0000000000b7e39f in timeout_next (base=0x15e72a0, tv_p=0x47d34068) at /mnt/x25m/gval/r3483/extra/libevent/event.c:824
#3 0x0000000000b7d90d in event_base_loop (base=0x15e72a0, flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:486
#4 0x0000000000b7d7fc in event_loop (flags=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:443
#5 0x00000000006afeed in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:592
#6 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#7 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 9 (Thread 5176):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 8 (Thread 5178):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 7 (Thread 5179):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 6 (Thread 5180):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 5 (Thread 5181):
#0 0x0000003fccc0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000008cf135 in safe_cond_timedwait (cond=0x12d86c8, mp=0x12d80e8, abstime=0x47e780e0, file=0xc29610 "/mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h", line=1009) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:278
#2 0x00000000007e60d2 in inline_mysql_cond_timedwait (that=0x12d86c8, mutex=0x12d80e8, abstime=0x47e780e0, src_file=0xc29038 "/mnt/x25m/gval/r3483/sql/log.cc", src_line=5533) at /mnt/x25m/gval/r3483/include/mysql/psi/mysql_thread.h:1009
#3 0x00000000007e6191 in MYSQL_BIN_LOG::wait_for_update_bin_log (this=0x12d80e0, thd=0x1d20460, timeout=0x47e780e0) at /mnt/x25m/gval/r3483/sql/log.cc:5532
#4 0x00000000005fbf8f in mysql_binlog_send (thd=0x1d20460, log_ident=0x2aaab0023180 "master-bin.000001", pos=170816, flags=0) at /mnt/x25m/gval/r3483/sql/sql_repl.cc:874
#5 0x00000000005dd528 in dispatch_command (command=COM_BINLOG_DUMP, thd=0x1d20460, packet=0x1d31f41 "", packet_length=27) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:1211
#6 0x00000000005de23d in do_command (thd=0x1d20460) at /mnt/x25m/gval/r3483/sql/sql_parse.cc:771
#7 0x00000000006b004d in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:649
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 4 (Thread 5182):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 3 (Thread 5184):
#0 0x0000003fccc0d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x0000003fccc08e50 in _L_lock_1233 () from /lib64/libpthread.so.0
#2 0x0000003fccc08dd3 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x00000000008cebc6 in safe_mutex_lock (mp=0x12c7f20, try_lock=0 '\000', file=0xbbb5e0 "/mnt/x25m/gval/r3483/sql/scheduler.cc", line=581) at /mnt/x25m/gval/r3483/mysys/thr_mutex.c:152
#4 0x00000000006afeb9 in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:581
#5 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#6 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 2 (Thread 5186):
#0 0x0000003fccc0e838 in do_sigwait () from /lib64/libpthread.so.0
#1 0x0000003fccc0e8dd in sigwait () from /lib64/libpthread.so.0
#2 0x0000000000538ce4 in signal_hand (arg=0x0) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2739
#3 0x0000000000b2a95d in pfs_spawn_thread (arg=0x15e75c0) at /mnt/x25m/gval/r3483/storage/perfschema/pfs.cc:1015
#4 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#5 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

Thread 1 (Thread 5185):
#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=11) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=11) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000000000b7e95f in min_heap_erase (s=0x15e74a8, e=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/min_heap.h:88
#5 0x0000000000b7e8ae in event_queue_remove (base=0x15e72a0, ev=0x2aaab0001070, queue=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:925
#6 0x0000000000b7e23f in event_del (ev=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/event.c:780
#7 0x00000000006aff5a in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:606
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

[Valentine Gostev (longbow)]

# bzr version-info
revision-id: <email address hidden>
date: 2011-05-31 11:33:25 +0300
build-date: 2011-05-31 14:34:47 -0700
revno: 3483

[Valentine Gostev (longbow)]

bt:

#0 0x0000003fccc0bd02 in pthread_kill () from /lib64/libpthread.so.0
#1 0x00000000008c8393 in my_write_core (sig=11) at /mnt/x25m/gval/r3483/mysys/stacktrace.c:423
#2 0x00000000005395a7 in handle_segfault (sig=11) at /mnt/x25m/gval/r3483/sql/mysqld.cc:2546
#3 <signal handler called>
#4 0x0000000000b7e95f in min_heap_erase (s=0x15e74a8, e=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/min_heap.h:88
#5 0x0000000000b7e8ae in event_queue_remove (base=0x15e72a0, ev=0x2aaab0001070, queue=1) at /mnt/x25m/gval/r3483/extra/libevent/event.c:925
#6 0x0000000000b7e23f in event_del (ev=0x2aaab0001070) at /mnt/x25m/gval/r3483/extra/libevent/event.c:780
#7 0x00000000006aff5a in libevent_thread_proc (arg=0x0) at /mnt/x25m/gval/r3483/sql/scheduler.cc:606
#8 0x0000003fccc0673d in start_thread () from /lib64/libpthread.so.0
#9 0x0000003fcc4d3f6d in clone () from /lib64/libc.so.6

[Valentine Gostev (longbow)]

core dump