Above fix works but another deadlock is hit: pthread_cond_wait@@GLIBC_2.3.2 os_cond_wait os_event_wait_low sync_array_wait_event mutex_spin_wait mutex_enter_func pfs_mutex_enter_func innobase_kill_connection -----------------------------> trx_mutex_enter deadlock kill_handlerton plugin_foreach_with_mask ha_kill_connection THD::awake wsrep_thd_awake wsrep_innobase_kill_one_trx wsrep_kill_victim lock_rec_other_has_conflicting lock_rec_lock_slow ---------------------------------------------> trx_mutex_enter lock_rec_lock lock_clust_rec_read_check_and_lock sel_set_rec_lock row_search_for_mysql ha_innobase::index_read handler::index_read_map handler::ha_index_read_map handler::read_range_first handler::multi_range_read_next DsMrr_impl::dsmrr_next ha_innobase::multi_range_read_next QUICK_RANGE_SELECT::get_next rr_quick mysql_update mysql_execute_command mysql_parse Query_log_event::do_apply_event Query_log_event::do_apply_event Log_event::apply_event wsrep_apply_rbr wsrep_apply_cb galera::TrxHandle::apply apply_trx_ws galera::ReplicatorSMM::apply_trx galera::ReplicatorSMM::process_trx galera::GcsActionSource::dispatch galera::GcsActionSource::process galera::ReplicatorSMM::async_recv galera_recv wsrep_replication_process start_wsrep_THD start_thread clone ------------------------------------------ The above fails as : InnoDB: sync levels should be > 299 but a level is 297 Mutex '&trx->mutex' InnoDB: Locked mutex: addr 0x7f4ad8018360 thread 139960774665984 file /mnt/workspace/build-xtradb-cluster-binaries-56/BUILD_TYPE/debug/label_exp/centos6-64/Percona-XtraDB-Cluster-5.6.13/storage/innobase/lock/lock0lock.cc line 2411 InnoDB: sync_thread_levels_g(array, 299) does not hold! 2013-09-30 22:15:57 7f4b2841a700 InnoDB: Assertion failure in thread 139960774665984 in file sync0sync.cc line 1250 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 19:15:57 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona Server better by reporting any bugs at http://bugs.percona.com/ key_buffer_size=1048576 read_buffer_size=131072 max_used_connections=11 max_threads=153 thread_count=11 connection_count=11 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 62039 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4ad80009a0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f4b28419d98 thread_stack 0x40000 /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(my_print_stacktrace+0x35)[0x9ac345] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(handle_fatal_signal+0x4c4)[0x69e1a4] /lib64/libpthread.so.0[0x3e78e0f500] /lib64/libc.so.6(gsignal+0x35)[0x3e78a328e5] /lib64/libc.so.6(abort+0x175)[0x3e78a340c5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xb368bf] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xb378d2] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa2630d] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0x5cd0d6] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z24plugin_foreach_with_maskP3THDPFcS0_PP13st_plugin_intPvEijS4_+0x204)[0x74cfc4] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN3THD5awakeENS_12killed_stateE+0x280)[0x701450] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(wsrep_thd_awake+0x147)[0x7026d7] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa18e0f] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa6134c] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa674f3] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa6c372] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xb0a10f] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xb0dade] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0xa1c792] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN7handler17ha_index_read_mapEPhPKhm16ha_rkey_function+0x144)[0x5c89d4] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN7handler16read_range_firstEPK12st_key_rangeS2_bb+0xa7)[0x5ca8c7] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN7handler21multi_range_read_nextEPPc+0xb5)[0x5c92b5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN10DsMrr_impl10dsmrr_nextEPPc+0x128)[0x5cb618] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN18QUICK_RANGE_SELECT8get_nextEv+0x7d)[0x87ffcd] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0x8a7189] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z12mysql_updateP3THDP10TABLE_LISTR4ListI4ItemES6_PS4_jP8st_ordery15enum_duplicatesbPySB_+0x1143)[0x7ba5f3] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z21mysql_execute_commandP3THD+0xd05)[0x73c9d5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x6f8)[0x743478] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN15Query_log_event14do_apply_eventEPK14Relay_log_infoPKcj+0x58d)[0x9256dd] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_ZN9Log_event11apply_eventEP14Relay_log_info+0x99)[0x92c559] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld[0x5c1ae3] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z14wsrep_apply_cbPvPKvmPK14wsrep_trx_meta+0xb5)[0x5c22b5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZNK6galera9TrxHandle5applyEPvPF15wsrep_cb_statusS1_PKvmPK14wsrep_trx_metaERS6_+0x12a)[0x7f4b2b7e8b7a] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(+0x1ad1e5)[0x7f4b2b8241e5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM9apply_trxEPvPNS_9TrxHandleE+0x273)[0x7f4b2b824ff3] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM11process_trxEPvPNS_9TrxHandleE+0x45)[0x7f4b2b8258b5] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZN6galera15GcsActionSource8dispatchEPvRK10gcs_actionRb+0x2cc)[0x7f4b2b7fd0ec] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZN6galera15GcsActionSource7processEPvRb+0x63)[0x7f4b2b7fd5c3] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(_ZN6galera13ReplicatorSMM10async_recvEPv+0x93)[0x7f4b2b81a563] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/lib/libgalera_smm.so(galera_recv+0x23)[0x7f4b2b834f13] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(_Z25wsrep_replication_processP3THD+0x5f)[0x73a60f] /mnt/workspace/PXC-5.6-msysbench/BTYPE/debug/Host/centos6-64/Percona-XtraDB-Cluster-5.6.13-24.0.463.Linux.x86_64/bin/mysqld(start_wsrep_THD+0x497)[0x5b3da7] /lib64/libpthread.so.0[0x3e78e07851] /lib64/libc.so.6(clone+0x6d)[0x3e78ae894d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f4ad8008eb0): UPDATE sbtest1 SET c='77398094050-97018391085-98721189222-66570166619-96064899306-27923164134-78251786109-31146758939-04091889884-69345270788' WHERE id=49605 Connection ID (thread ID): 3 Status: NOT_KILLED