rsync SST not working under load

[Teemu Ollakka]

Under dbt2 rsync SST causes inconsistency on joiner and under sysbench it causes a crash:

110607 14:51:05 [Note] WSREP: Shifting JOINER -> JOINED (TO: 4283316)
110607 14:54:24 [ERROR] Slave SQL: Could not execute Update_rows event on table sbtest.sbtest; Can't find record in 'sbtest', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 709, Error_code: 1032
110607 14:54:24 [Warning] WSREP: RBR event 4 apply warning: 120, 4235189
110607 14:54:26 [ERROR] WSREP: failed to apply trx: source: d8bd5808-90fa-11e0-0800-551188b70587 state: CERTIFYING flags: 1 conn_id: 30 trx_id: 8497344 seqnos (l: 58251, g: 4235189, s: 4235178, d: 4235157, ts: 1307447418791755908)
110607 14:54:26 [ERROR] WSREP: Failed to apply app buffer: z�M, seqno: 4235189, status: WSREP_FATAL

Examination of dbt2 dump diffs suggests that a number of transactions are missing on the joiner after rsync SST (it is missing rows and has older timestamps).

Dbt2 diff attached.

[Teemu Ollakka (teemu-ollakka)]

After making SST from slave, got the following on joiner:

InnoDB: Starting in background the rollback of uncommitted transactions
110607 22:13:36 InnoDB: Rolling back trx with id 0 9564622, 1 rows to undo

InnoDB: Rolling back of trx id 0 9564622 completed

However, there should be zero uncommitted transactions on slave in DONOR state.