Comment 2 for bug 1237816

Hi,

Our Bugzilla app is on a 3 node multi-master cluster and having the same issue. Can you provide a bit more understanding as to why this happens?

131231 18:16:43 [ERROR] Slave SQL: Could not execute Update_rows event on table bugzilla.attachments; Can't find record in 'attachments', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 279, Error_code: 1032
131231 18:16:43 [Warning] WSREP: RBR event 2 Update_rows apply warning: 120, 1593958
131231 18:16:43 [Note] WSREP: declaring 62a92d9f-7179-11e3-b4ca-de001387c324 stable
131231 18:16:43 [ERROR] WSREP: Failed to apply trx: source: 62a92d9f-7179-11e3-b4ca-de001387c324 version: 2 local: 0 state: APPLYING flags: 1 conn_id: 32030272 trx_id: 8936786 seqnos (l: 550435, g: 1593958, s: 1593957, d: 1593718, ts: 1388535403140834735)
131231 18:16:43 [ERROR] WSREP: Failed to apply app buffer: seqno: 1593958, status: WSREP_FATAL
         at galera/src/replicator_smm.cpp:apply_wscoll():52
         at galera/src/replicator_smm.cpp:apply_trx_ws():118
131231 18:16:43 [ERROR] WSREP: Node consistency compromized, aborting...

On the front, there is a 4 node web cluster serving BZ. I had made a config change and restarted httpd on each node when the inconsistency occurred. I can't help to think that was a contributor.

MariaDB-Galera-server-5.5.34-1.x86_64
wsrep_provider_name Galera
wsrep_provider_vendor Codership Oy <email address hidden>
wsrep_provider_version 23.2.7(r157)

wsrep_replicate_myisam ON (all BZ tables InnoDB)
wsrep_retry_autocommit 1
wsrep_slave_threads 1
wsrep_sst_auth
wsrep_sst_donor
wsrep_sst_donor_rejects_queries OFF
wsrep_sst_method rsync
wsrep_sst_receive_address AUTO

Is this caused by some app race condition some other factor? I am also load balancing :3306 with haproxy on each of the 4 web nodes. haproxy.cfg:

listen mysql_proxy 127.0.0.1:3306
   timeout connect 5s
   timeout client 1m
   timeout server 1m
   mode tcp
   balance roundrobin
   option tcpka
   option mysql-check user haproxy
   server web-db1 10.0.0.1:3306 check weight 1
   server web-db2 10.0.0.2:3306 check weight 1
   server web-db3 10.0.0.3:3306 check weight 1

Thank you.