mysql crash

[Ammon Sutherland]

CentOS 6.3, fully patched as of 2/27/2013
Percona rhel6 rpm's for 5.5.29-rel29.4.401
server is a slave, DB Size = 2.8TB, 95% innodb, 5% myisam.

This server crashes roughly every 2-5 days with the following stack trace from the error log:

130226 14:12:11 InnoDB: Assertion failure in thread 140352895031040 in file buf0buf.c line 2608
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
20:12:11 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona Server better by reporting any
bugs at http://bugs.percona.com/

key_buffer_size=4294967296
read_buffer_size=262144
max_used_connections=284
max_threads=3000
thread_count=95
connection_count=95
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 5767077 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7fa65b8f6000
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7fa674738e08 thread_stack 0x40000
/usr/sbin/mysqld(my_print_stacktrace+0x35)[0x7ac7e5]
/usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x688084]
/lib64/libpthread.so.0(+0xf500)[0x7fb77bd2b500]
/lib64/libc.so.6(gsignal+0x35)[0x7fb77aef38a5]
/lib64/libc.so.6(abort+0x175)[0x7fb77aef5085]
/usr/sbin/mysqld[0x8affc5]
/usr/sbin/mysqld[0x89dd8b]
/usr/sbin/mysqld[0x84d3d7]
/usr/sbin/mysqld[0x84de69]
/usr/sbin/mysqld[0x820b78]
/usr/sbin/mysqld(_ZN7handler18index_read_idx_mapEPhjPKhm16ha_rkey_function+0x78)[0x688bb8]
/usr/sbin/mysqld[0x5aef19]
/usr/sbin/mysqld[0x5b4a16]
/usr/sbin/mysqld[0x5be56c]
/usr/sbin/mysqld(_ZN4JOIN8optimizeEv+0x52c)[0x5c02ec]
/usr/sbin/mysqld(_Z12mysql_selectP3THDPPP4ItemP10TABLE_LISTjR4ListIS1_ES2_jP8st_orderSB_S2_SB_yP13select_resultP18st_select_lex_unitP13st_select_lex+0xd1)[0x5c91b1]
/usr/sbin/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x1cd)[0x5c9cbd]
/usr/sbin/mysqld[0x586472]
/usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x1236)[0x58aab6]
/usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x333)[0x58e483]
/usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x15cd)[0x58fafd]
/usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0xd7)[0x62a397]
/usr/sbin/mysqld(handle_one_connection+0x51)[0x62a4d1]
/usr/sbin/mysqld(pfs_spawn_thread+0x59)[0x810819]
/lib64/libpthread.so.0(+0x7851)[0x7fb77bd23851]
/lib64/libc.so.6(clone+0x6d)[0x7fb77afa911d]

InnoDB Options in the config file:

innodb_file_format=barracuda
innodb_buffer_pool_size = 58G
innodb_additional_mem_pool_size = 1G
innodb_log_group_home_dir=/var/lib/mysql_logs
innodb_log_buffer_size = 128M
innodb_log_file_size = 1G
innodb_flush_log_at_trx_commit = 2
innodb_thread_concurrency = 16
innodb_flush_method = ALL_O_DIRECT
innodb_file_per_table = 1
innodb_sync_spin_loops=200
innodb_stats_on_metadata=0
innodb_stats_method=nulls_unequal
innodb_stats_sample_pages=32
innodb_io_capacity=2000
innodb_open_files=8000
innodb_adaptive_flushing=1
innodb_read_io_threads=4
innodb_write_io_threads=4
innodb_autoinc_lock_mode=2
innodb_stats_update_need_lock=0
expand_fast_index_creation=1
innodb_old_blocks_time=1000
innodb_concurrency_tickets=1000
innodb_fast_checksum=1
innodb_log_block_size=4096
innodb_lock_wait_timeout=180
transaction-isolation = READ-COMMITTED
innodb_use_sys_stats_table=1
innodb_lazy_drop_table=1
innodb_purge_threads=1
innodb_buffer_pool_instances=2
innodb_large_prefix=1
innodb_flush_neighbor_pages=cont
innodb_read_ahead_threshold=48
innodb_purge_batch_size=100
default_storage_engine=MyISAM

[Laurynas Biveinis (laurynas-biveinis)]

Thank you for your bug report. The crash looks identical to bug 1086700, thus I'm going to close this one as its duplicate. Please track that bug and add any comments there.