Yasufumi,

It does not seem hardware bug, I have no problem with standard Innodb.

Also why you think it is related to page_hash. As you see show innodb
status also hang, there is mutex srv_monitor_mutex affected. It seems
sync_array is in problem place.





On 2/9/09, Yasufumi Kinoshita <email address hidden> wrote:
> I think there may be probability of hardware bug(?) or timing problem of
> memory access.
> But we should consider such problem.
>
> I will implement some safety code.
>
>
> (*) page_hash_latch is not locked, but x_lock is still waiting.
>
> lock_word = 1048576 ( unlocked ), reader_count = 0, writer = 351 (
> RW_LOCK_EXCLUSIVE ),
> writer_thread = 1204177232 ( 47c64950 ) , writer_count = 0, pass = 0,
> s_waiters = 1,
> x_waiters = 0, wait_ex_waiters = 1,
> last_s_file_name = 0x8c413a "btr/btr0sea.c",
> last_x_file_name = 0x8c5f16 "buf/buf0lru.c", writer_is_wait_ex = 1,
> last_s_line = 821, last_x_line = 1450,
>
> --
> XtraDB hangs in 10 threads TPC-C
> https://bugs.launchpad.net/bugs/326445
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in Percona XtraDB Storage Engine for MySQL: New
>
> Bug description:
> (gdb) info threads
>   20 Thread 0x40a2f950 (LWP 13698)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   19 Thread 0x41776950 (LWP 13699)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   18 Thread 0x41f77950 (LWP 13700)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   17 Thread 0x42778950 (LWP 13701)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   16 Thread 0x4377a950 (LWP 13721)  0x00007ffdc811c482 in select () from
> /lib/libc.so.6
>   15 Thread 0x43f7b950 (LWP 13722)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   14 Thread 0x4477c950 (LWP 13723)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   13 Thread 0x447bd950 (LWP 13724)  0x00007ffdc93d4e7b in sigwait () from
> /lib/libpthread.so.0
>   12 Thread 0x40a70950 (LWP 13761)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   11 Thread 0x447fe950 (LWP 13763)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   10 Thread 0x4483f950 (LWP 13766)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   9 Thread 0x44880950 (LWP 13768)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   8 Thread 0x448c1950 (LWP 13770)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   7 Thread 0x44902950 (LWP 13771)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   6 Thread 0x44943950 (LWP 13772)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   5 Thread 0x44984950 (LWP 13773)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   4 Thread 0x449c5950 (LWP 13774)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   3 Thread 0x44a06950 (LWP 13775)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   2 Thread 0x40c8c950 (LWP 15994)  0x00007ffdc93d12d9 in
> pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
>   1 Thread 0x7ffdc97f46f0 (LWP 13695)  0x00007ffdc811c482 in select () from
> /lib/libc.so.6
>
>
> (gdb) thread apply all backtrace
>
> Thread 20 (Thread 0x40a2f950 (LWP 13698)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x0000000000794123 in os_aio_simulated_handle ()
> #3  0x000000000075db3f in fil_aio_wait ()
> #4  0x00000000007d5540 in io_handler_thread ()
> #5  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #6  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #7  0x0000000000000000 in ?? ()
>
> Thread 19 (Thread 0x41776950 (LWP 13699)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x0000000000794123 in os_aio_simulated_handle ()
> #3  0x000000000075db3f in fil_aio_wait ()
> #4  0x00000000007d5540 in io_handler_thread ()
> #5  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #6  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #7  0x0000000000000000 in ?? ()
>
> Thread 18 (Thread 0x41f77950 (LWP 13700)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x0000000000794123 in os_aio_simulated_handle ()
> #3  0x000000000075db3f in fil_aio_wait ()
> #4  0x00000000007d5540 in io_handler_thread ()
> #5  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #6  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #7  0x0000000000000000 in ?? ()
>
> Thread 17 (Thread 0x42778950 (LWP 13701)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d97a2 in mutex_spin_wait ()
> #4  0x000000000073c1cc in buf_page_io_complete ()
> #5  0x000000000075dcb7 in fil_aio_wait ()
> #6  0x00000000007d5540 in io_handler_thread ()
> #7  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #8  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #9  0x0000000000000000 in ?? ()
>
> Thread 16 (Thread 0x4377a950 (LWP 13721)):
> #0  0x00007ffdc811c482 in select () from /lib/libc.so.6
> #1  0x0000000000797adb in os_thread_sleep ()
> #2  0x00000000007d4435 in srv_lock_timeout_and_monitor_thread ()
> #3  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #4  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #5  0x0000000000000000 in ?? ()
>
> Thread 15 (Thread 0x43f7b950 (LWP 13722)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d97a2 in mutex_spin_wait ()
> #4  0x00000000007d31eb in srv_error_monitor_thread ()
> ---Type <return> to continue, or q <return> to quit---
> #5  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #6  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #7  0x0000000000000000 in ?? ()
>
> Thread 14 (Thread 0x4477c950 (LWP 13723)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d97a2 in mutex_spin_wait ()
> #4  0x000000000078607b in log_buffer_flush_to_disk ()
> #5  0x00000000007d22b1 in srv_master_thread ()
> #6  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #7  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #8  0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 0x447bd950 (LWP 13724)):
> #0  0x00007ffdc93d4e7b in sigwait () from /lib/libpthread.so.0
> #1  0x000000000059e45b in signal_hand ()
> #2  0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #3  0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 0x40a70950 (LWP 13761)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007778fd in ibuf_insert_low ()
> #6  0x0000000000778ad5 in ibuf_insert ()
> #7  0x000000000072ccf8 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> #9  0x00000000007b4bc7 in row_ins_step ()
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 0x447fe950 (LWP 13763)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007778fd in ibuf_insert_low ()
> #6  0x0000000000778ad5 in ibuf_insert ()
> ---Type <return> to continue, or q <return> to quit---
> #7  0x000000000072ccf8 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> #9  0x00000000007b4bc7 in row_ins_step ()
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 0x4483f950 (LWP 13766)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x0000000000742732 in buf_flush_buffered_writes ()
> #3  0x0000000000743bea in buf_flush_batch ()
> #4  0x00000000007444fa in buf_flush_free_margin ()
> #5  0x0000000000749e8f in buf_read_page ()
> #6  0x000000000073ee3d in buf_page_get_gen ()
> #7  0x000000000072ca0e in btr_cur_search_to_nth_level ()
> #8  0x00000000007c61b3 in row_search_for_mysql ()
> #9  0x000000000071ceda in ha_innobase::index_read ()
> #10 0x0000000000686609 in handler::index_read_idx_map ()
> #11 0x00000000005ffb65 in join_read_const ()
> #12 0x000000000060790d in join_read_const_table ()
> #13 0x000000000060cdfe in make_join_statistics ()
> #14 0x000000000060e7db in JOIN::optimize ()
> #15 0x0000000000611d40 in mysql_select ()
> #16 0x000000000061754c in handle_select ()
> #17 0x00000000005abecd in execute_sqlcom_select ()
> #18 0x00000000005afbc3 in mysql_execute_command ()
> #19 0x0000000000622cca in Prepared_statement::execute ()
> #20 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #21 0x0000000000626365 in mysql_stmt_execute ()
> #22 0x00000000005b482e in dispatch_command ()
> #23 0x00000000005b5ad6 in do_command ()
> #24 0x00000000005a8646 in handle_one_connection ()
> #25 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #26 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #27 0x0000000000000000 in ?? ()
>
> Thread 9 (Thread 0x44880950 (LWP 13768)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x000000000072ca0e in btr_cur_search_to_nth_level ()
> #6  0x00000000007c5f78 in row_search_for_mysql ()
> ---Type <return> to continue, or q <return> to quit---
> #7  0x000000000071ceda in ha_innobase::index_read ()
> #8  0x0000000000686609 in handler::index_read_idx_map ()
> #9  0x00000000005ffb65 in join_read_const ()
> #10 0x000000000060790d in join_read_const_table ()
> #11 0x000000000060cdfe in make_join_statistics ()
> #12 0x000000000060e7db in JOIN::optimize ()
> #13 0x0000000000611d40 in mysql_select ()
> #14 0x000000000061754c in handle_select ()
> #15 0x00000000005abecd in execute_sqlcom_select ()
> #16 0x00000000005afbc3 in mysql_execute_command ()
> #17 0x0000000000622cca in Prepared_statement::execute ()
> #18 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #19 0x0000000000626365 in mysql_stmt_execute ()
> #20 0x00000000005b482e in dispatch_command ()
> #21 0x00000000005b5ad6 in do_command ()
> #22 0x00000000005a8646 in handle_one_connection ()
> #23 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #24 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #25 0x0000000000000000 in ?? ()
>
> Thread 8 (Thread 0x448c1950 (LWP 13770)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007778fd in ibuf_insert_low ()
> #6  0x0000000000778ad5 in ibuf_insert ()
> #7  0x000000000072ccf8 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> #9  0x00000000007b4bc7 in row_ins_step ()
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 7 (Thread 0x44902950 (LWP 13771)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007778fd in ibuf_insert_low ()
> #6  0x0000000000778ad5 in ibuf_insert ()
> #7  0x000000000072ccf8 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> ---Type <return> to continue, or q <return> to quit---
> #9  0x00000000007b4bc7 in row_ins_step ()
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 6 (Thread 0x44943950 (LWP 13772)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d936f in rw_lock_x_lock_func ()
> #4  0x0000000000747a9b in buf_LRU_free_block ()
> #5  0x0000000000748abe in buf_LRU_search_and_free_block ()
> #6  0x0000000000748eea in buf_LRU_try_free_flushed_blocks ()
> #7  0x000000000072cf85 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> #9  0x00000000007b4bc7 in row_ins_step ()
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 5 (Thread 0x44984950 (LWP 13773)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d97a2 in mutex_spin_wait ()
> #4  0x0000000000785f76 in log_close ()
> #5  0x000000000079162c in mtr_commit ()
> #6  0x00000000007ca9ce in row_upd_clust_rec ()
> #7  0x00000000007ce410 in row_upd_clust_step ()
> #8  0x00000000007cee6b in row_upd_step ()
> #9  0x00000000007ba416 in row_update_for_mysql ()
> #10 0x000000000071ff4b in ha_innobase::update_row ()
> #11 0x000000000068a7c8 in handler::ha_update_row ()
> ---Type <return> to continue, or q <return> to quit---
> #12 0x000000000062aac8 in mysql_update ()
> #13 0x00000000005afa84 in mysql_execute_command ()
> #14 0x0000000000622cca in Prepared_statement::execute ()
> #15 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #16 0x0000000000626365 in mysql_stmt_execute ()
> #17 0x00000000005b482e in dispatch_command ()
> #18 0x00000000005b5ad6 in do_command ()
> #19 0x00000000005a8646 in handle_one_connection ()
> #20 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #21 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #22 0x0000000000000000 in ?? ()
>
> Thread 4 (Thread 0x449c5950 (LWP 13774)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007721e5 in ibuf_bitmap_get_map_page ()
> #6  0x0000000000779702 in ibuf_merge_or_delete_for_page ()
> #7  0x000000000073c237 in buf_page_io_complete ()
> #8  0x00000000007495f7 in buf_read_page_low ()
> #9  0x0000000000749e76 in buf_read_page ()
> #10 0x000000000073ee3d in buf_page_get_gen ()
> #11 0x000000000072ca0e in btr_cur_search_to_nth_level ()
> #12 0x00000000007c61b3 in row_search_for_mysql ()
> #13 0x000000000071ceda in ha_innobase::index_read ()
> #14 0x0000000000686609 in handler::index_read_idx_map ()
> #15 0x00000000005ffb65 in join_read_const ()
> #16 0x000000000060790d in join_read_const_table ()
> #17 0x000000000060cdfe in make_join_statistics ()
> #18 0x000000000060e7db in JOIN::optimize ()
> #19 0x0000000000611d40 in mysql_select ()
> #20 0x000000000061754c in handle_select ()
> #21 0x00000000005abecd in execute_sqlcom_select ()
> #22 0x00000000005afbc3 in mysql_execute_command ()
> #23 0x0000000000622cca in Prepared_statement::execute ()
> #24 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #25 0x0000000000626365 in mysql_stmt_execute ()
> #26 0x00000000005b482e in dispatch_command ()
> #27 0x00000000005b5ad6 in do_command ()
> #28 0x00000000005a8646 in handle_one_connection ()
> #29 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #30 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #31 0x0000000000000000 in ?? ()
>
> Thread 3 (Thread 0x44a06950 (LWP 13775)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d88ca in rw_lock_s_lock_spin ()
> #4  0x000000000073e16f in buf_page_get_gen ()
> #5  0x00000000007778fd in ibuf_insert_low ()
> #6  0x0000000000778ad5 in ibuf_insert ()
> #7  0x000000000072ccf8 in btr_cur_search_to_nth_level ()
> #8  0x00000000007b3766 in row_ins_index_entry_low ()
> #9  0x00000000007b4bc7 in row_ins_step ()
> ---Type <return> to continue, or q <return> to quit---
> #10 0x00000000007bb656 in row_insert_for_mysql ()
> #11 0x000000000072095e in ha_innobase::write_row ()
> #12 0x000000000068a8a5 in handler::ha_write_row ()
> #13 0x0000000000619abc in write_record ()
> #14 0x000000000061d865 in mysql_insert ()
> #15 0x00000000005af4bb in mysql_execute_command ()
> #16 0x0000000000622cca in Prepared_statement::execute ()
> #17 0x0000000000625c8c in Prepared_statement::execute_loop ()
> #18 0x0000000000626365 in mysql_stmt_execute ()
> #19 0x00000000005b482e in dispatch_command ()
> #20 0x00000000005b5ad6 in do_command ()
> #21 0x00000000005a8646 in handle_one_connection ()
> #22 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #23 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 0x40c8c950 (LWP 15994)):
> #0  0x00007ffdc93d12d9 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/libpthread.so.0
> #1  0x0000000000796f8d in os_event_wait_low ()
> #2  0x00000000007d811c in sync_array_wait_event ()
> #3  0x00000000007d97a2 in mutex_spin_wait ()
> #4  0x00000000007864b3 in log_print ()
> #5  0x00000000007d3bb3 in srv_printf_innodb_monitor ()
> #6  0x000000000071e076 in innodb_show_status ()
> #7  0x000000000068c417 in ha_show_status ()
> #8  0x00000000005af951 in mysql_execute_command ()
> #9  0x00000000005b43d7 in mysql_parse ()
> #10 0x00000000005b4f3b in dispatch_command ()
> #11 0x00000000005b5ad6 in do_command ()
> #12 0x00000000005a8646 in handle_one_connection ()
> #13 0x00007ffdc93cd3ea in start_thread () from /lib/libpthread.so.0
> #14 0x00007ffdc8123c6d in clone () from /lib/libc.so.6
> #15 0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 0x7ffdc97f46f0 (LWP 13695)):
> #0  0x00007ffdc811c482 in select () from /lib/libc.so.6
> #1  0x00000000005a2b4d in handle_connections_sockets ()
> #2  0x00000000005a36e8 in main ()
> #0  0x00007ffdc811c482 in select () from /lib/libc.so.6
>