Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC

WSREP: trx_replay failed for: 6, query: void

Bug #1316769 reported by Travis Glenn Hansen on 2014-05-06

This bug report is a duplicate of: Bug #1327763: Random crash on the same node. Edit Remove

This bug affects 2 people

	Status	Importance	Assigned to
MySQL patches by Codership	Status tracked in 5.6
5.5	New	Undecided	Unassigned
5.6	New	Undecided	Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC	New	Undecided	Unassigned

Bug Description

We regularly but sporadically are having nodes drop in the cluster since upgrading to 5.6. Current environment is:

1. CentOS6 fully updated (same across all nodes)

[root@lava ~]# rpm -qa | grep -i percona
percona-xtrabackup-2.1.8-733.rhel6.x86_64
Percona-XtraDB-Cluster-galera-3-3.4-1.213.rhel6.x86_64
Percona-XtraDB-Cluster-shared-56-5.6.15-25.5.759.rhel6.x86_64
Percona-XtraDB-Cluster-56-5.6.15-25.5.759.rhel6.x86_64
percona-toolkit-2.2.7-1.noarch
Percona-Server-shared-51-5.1.73-rel14.11.603.rhel6.x86_64
Percona-XtraDB-Cluster-client-56-5.6.15-25.5.759.rhel6.x86_64
Percona-XtraDB-Cluster-server-56-5.6.15-25.5.759.rhel6.x86_64
percona-release-0.0-1.x86_64

We've been running with both 3 and 4 nodes in the cluster and experiencing the same thing.

Here is error log output:

2014-05-06 12:52:38 39198 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 170344978)
2014-05-06 12:52:38 39198 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 170344978
2014-05-06 12:52:38 39198 [Warning] WSREP: Failed to apply app buffer: seqno: 170344978, status: 1
  at galera/src/trx_handle.cpp:apply():340
Retrying 2th time
2014-05-06 12:52:38 39198 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 170344978)
2014-05-06 12:52:38 39198 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 170344978
2014-05-06 12:52:38 39198 [Warning] WSREP: Failed to apply app buffer: seqno: 170344978, status: 1
  at galera/src/trx_handle.cpp:apply():340
Retrying 3th time
2014-05-06 12:52:38 39198 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 170344978)
2014-05-06 12:52:38 39198 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 170344978
2014-05-06 12:52:38 39198 [Warning] WSREP: Failed to apply app buffer: seqno: 170344978, status: 1
  at galera/src/trx_handle.cpp:apply():340
Retrying 4th time
2014-05-06 12:52:38 39198 [Warning] WSREP: BF applier failed to open_and_lock_tables: 1615, fatal: 0 wsrep = (exec_mode: 1 conflict_state: 5 seqno: 170344978)
2014-05-06 12:52:38 39198 [Warning] WSREP: RBR event 3 Delete_rows apply warning: 1615, 170344978
2014-05-06 12:52:38 39198 [Warning] WSREP: failed to replay trx: source: 1dc9e882-c298-11e3-8455-4612c5d10e02 version: 3 local: 1 state: REPLAYING flags: 1 conn_id: 4115970 trx_id: 6786021497 seqnos (l: 12597015, g: 170344978, s: 170344975, d: 170344882, ts: 2403522400037087)
2014-05-06 12:52:38 39198 [Warning] WSREP: Failed to apply trx 170344978 4 times
2014-05-06 12:52:38 39198 [ERROR] WSREP: trx_replay failed for: 6, query: void
2014-05-06 12:52:38 39198 [ERROR] Aborting