The seriees of events based on the logs:
1. node3 dropped out of cluster
2. node2 splits, goes non-primary
3. node1 splits, goes non-primary
4. node2 and node1 found each other, remained non-primary while waiting for node3 to come back
I would expect at #4 above, the cluster would've have regained quorum and switched to PRIMARY but it did not.
Relevant logs from node2:
140114 20:32:22 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.2.9:4567
140114 20:32:22 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 0
140114 20:32:22 [Warning] WSREP: last inactive check more than PT1.5S ago (PT2.38318S), skipping check
140114 20:32:23 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:23 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:24 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:24 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 381d72fa-4b83-11e3-8d48-02e7ece87890 (tcp://192.168.2.102:4567), attempt 0
140114 20:32:24 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 381d72fa-4b83-11e3-8d48-02e7ece87890
140114 20:32:25 [Note] WSREP: evs::proto(ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, GATHER, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:26 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,309) memb {
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
381d72fa-4b83-11e3-8d48-02e7ece87890,
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:26 [Note] WSREP: view(view_id(NON_PRIM,ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,310) memb {
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
381d72fa-4b83-11e3-8d48-02e7ece87890,
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:26 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:26 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:26 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 441177589)
140114 20:32:26 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:26 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:26 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:26 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:27 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:27 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:35 [Note] WSREP: declaring 381d72fa-4b83-11e3-8d48-02e7ece87890 stable
140114 20:32:35 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,311) memb {
381d72fa-4b83-11e3-8d48-02e7ece87890,
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2
140114 20:32:35 [Note] WSREP: Flow-control interval: [23, 23]
140114 20:32:35 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:35 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 2, my index: 1, protocol version 2
140114 20:32:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:34:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 30
140114 20:36:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 60
140114 20:38:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 90
140114 20:40:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 120
140114 20:42:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 150
140114 20:44:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 180
140114 20:46:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 210
140114 20:48:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 240
140114 20:50:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 270
140114 20:52:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 300
140114 20:54:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 330
140114 20:56:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 360
140114 20:58:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 390
140114 21:00:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 420
140114 21:02:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 450
140114 21:04:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 480
140114 21:06:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 510
140114 21:08:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 540
140114 21:10:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 570
140114 21:12:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 600
140114 21:14:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 630
140114 21:16:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 660
140114 21:18:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 690
140114 21:20:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 720
140114 21:22:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 750
140114 21:24:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 780
140114 21:26:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 810
140114 21:28:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 840
140114 21:30:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 870
140114 21:32:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 900
140114 21:34:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 930
140114 21:36:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 960
140114 21:38:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 990
140114 21:40:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1020
140114 21:42:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1050
140114 21:44:23 [Note] WSREP: (ade80c49-54b9-11e3-81f2-2bdc4faa4ea7, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1080
Relevant logs from node1:
140114 2:53:48 [Warning] IP address '210.5.27.49' could not be resolved: Name or service not known
140114 20:32:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://192.168.2.9:4567
140114 20:32:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 0
140114 20:32:33 [Warning] WSREP: last inactive check more than PT1.5S ago (PT12.7579S), skipping check
140114 20:32:33 [Note] WSREP: evs::proto(381d72fa-4b83-11e3-8d48-02e7ece87890, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) detected inactive node: 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0
140114 20:32:33 [Note] WSREP: evs::proto(381d72fa-4b83-11e3-8d48-02e7ece87890, OPERATIONAL, view_id(REG,381d72fa-4b83-11e3-8d48-02e7ece87890,309)) suspecting node: ade80c49-54b9-11e3-81f2-2bdc4faa4ea7
140114 20:32:34 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,309) memb {
381d72fa-4b83-11e3-8d48-02e7ece87890,
} joined {
} left {
} partitioned {
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
})
140114 20:32:34 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,310) memb {
381d72fa-4b83-11e3-8d48-02e7ece87890,
} joined {
} left {
} partitioned {
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
})
140114 20:32:34 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:34 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:34 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:34 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 441177589)
140114 20:32:34 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1
140114 20:32:34 [Note] WSREP: Flow-control interval: [16, 16]
140114 20:32:34 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a55bea50, 394, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a4a47590, 394, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f49a50bf5c0, 396, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Warning] WSREP: Send action {0x7f4949ccc860, 735, TORDERED} returned -107 (Transport endpoint is not connected)
140114 20:32:34 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:34 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 2
140114 20:32:34 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:32:35 [Note] WSREP: declaring ade80c49-54b9-11e3-81f2-2bdc4faa4ea7 stable
140114 20:32:35 [Note] WSREP: view(view_id(NON_PRIM,381d72fa-4b83-11e3-8d48-02e7ece87890,311) memb {
381d72fa-4b83-11e3-8d48-02e7ece87890,
ade80c49-54b9-11e3-81f2-2bdc4faa4ea7,
} joined {
} left {
} partitioned {
6e2cdeb0-771a-11e3-ab2d-7603e7173ed0,
})
140114 20:32:35 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 2
140114 20:32:35 [Note] WSREP: Flow-control interval: [23, 23]
140114 20:32:35 [Note] WSREP: Received NON-PRIMARY.
140114 20:32:35 [Note] WSREP: New cluster view: global state: 7ebeccd0-4b67-11e3-8ce6-3308026f4bae:441177589, view# -1: non-Primary, number of nodes: 2, my index: 0, protocol version 2
140114 20:32:35 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
140114 20:34:36 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 30
140114 20:36:36 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 60
140114 20:38:38 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 90
140114 20:40:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 120
140114 20:42:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 150
140114 20:44:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 180
140114 20:46:40 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 210
140114 20:48:41 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 240
140114 20:50:43 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 270
140114 20:52:46 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 300
140114 20:54:46 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 330
140114 20:56:51 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 360
140114 20:58:53 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 390
140114 21:00:57 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 420
140114 21:03:00 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 450
140114 21:05:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 480
140114 21:07:17 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 510
140114 21:09:27 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 540
140114 21:11:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 570
140114 21:13:47 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 600
140114 21:16:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 630
140114 21:18:16 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 660
140114 21:20:28 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 690
140114 21:22:39 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 720
140114 21:24:52 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 750
140114 21:27:07 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 780
140114 21:29:21 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 810
140114 21:31:35 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 840
140114 21:33:48 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 870
140114 21:36:02 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 900
140114 21:38:18 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 930
140114 21:40:33 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 960
140114 21:42:48 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 990
140114 21:45:01 [Note] WSREP: (381d72fa-4b83-11e3-8d48-02e7ece87890, 'tcp://0.0.0.0:4567') reconnecting to 6e2cdeb0-771a-11e3-ab2d-7603e7173ed0 (tcp://192.168.2.9:4567), attempt 1020
~# dpkg -l|grep percona
ii percona-nagios-plugins 1.1.1-1 Percona Monitoring Plugins for Nagios
ii percona-toolkit 2.2.5-2 Advanced MySQL and system command-line tools
ii percona-xtrabackup 2.1.5-680-1.precise Open source backup tool for InnoDB and XtraDB
ii percona-xtradb-cluster-client-5.5 5.5.34-23.7.6-565.precise Percona Server database client binaries
ii percona-xtradb-cluster-common-5.5 5.5.34-23.7.6-565.precise Percona Server database common files (e.g. /etc/mysql/my.cnf)
ii percona-xtradb-cluster-galera-2.x 162.precise Galera components of Percona XtraDB Cluster
ii percona-xtradb-cluster-server-5.5 5.5.34-23.7.6-565.precise Percona Server database server binaries
ii xtrabackup 2.1.3-608.precise Transitional package for percona-xtrabackup
It turns out to be an expected behaviour due to safety considerations. It was speculated that in the case of a total cluster split:
{1,2,3} -> (1),(2),(3)
there is a chance of operator bootstrapping node 1 into a primary component. In that case merging and recovering primary component by simple majority
(2),(3) -> {2,3}
will lead to a second PC and state divergence.
Also (and it is not a speculation) in a busy cluster a split like that may lead to one node to be a bit more updated than the others. So complete remerge of the cluster
(1),(2),(3) -> {1,2,3}
is essential here for PC recovery due to consistency considerations.