Digging further, it does appear that the initially bootstrapped master is broken somehow: ``` 2023-05-07 15:06:46 0 [Warning] WSREP: Quorum: No node with complete state: Version : 6 Flags : 0x1 Protocols : 2 / 10 / 4 State : NON-PRIMARY Desync count : 0 Prim state : NON-PRIMARY Prim UUID : 00000000-0000-0000-0000-000000000000 Prim seqno : -1 First seqno : -1 Last seqno : -1 Commit cut : -1 Last vote : -1.0 Vote policy : 0 Prim JOINED : 0 State UUID : c91ea93c-ece8-11ed-bcbd-d37542ae8d5e Group UUID : 00000000-0000-0000-0000-000000000000 Name : 'mgr02' Incoming addr: '10.217.122.12:3306' Version : 6 Flags : 0x2 Protocols : 2 / 10 / 4 State : NON-PRIMARY Desync count : 0 Prim state : SYNCED Prim UUID : 5d5b432a-ece8-11ed-9285-be6bf8d043bf Prim seqno : 4 First seqno : 1 Last seqno : 4 Commit cut : 0 Last vote : -1.0 Vote policy : 0 Prim JOINED : 1 State UUID : c91ea93c-ece8-11ed-bcbd-d37542ae8d5e Group UUID : 78a25580-ece7-11ed-863d-5af2d3394150 Name : 'mgr00' Incoming addr: '10.217.122.10:3306' ``` ^^ is what happens after the replication error on initial bootstrap of a slave, and that slave is restarted The donor, being restarted after the two slaves fail to come up, is aware of them but can't figure out which one is primary: ``` 023-05-07 20:47:51 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6f57006e-ed18-11ed-9165-5f46e7399b53 from 1 (mgr00) 2023-05-07 20:47:51 0 [Warning] WSREP: Quorum: No node with complete state: Version : 6 Flags : 0x1 Protocols : 2 / 10 / 4 State : NON-PRIMARY Desync count : 0 Prim state : PRIMARY Prim UUID : cc9462f9-ed16-11ed-8544-d3da5d985617 Prim seqno : 5 First seqno : -1 Last seqno : 5 Commit cut : 0 Last vote : -1.0 Vote policy : 0 Prim JOINED : 1 State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53 Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a Name : 'mgr01' Incoming addr: '10.217.122.11:3306' Version : 6 Flags : 00 Protocols : 2 / 10 / 4 State : NON-PRIMARY Desync count : 0 Prim state : NON-PRIMARY Prim UUID : 00000000-0000-0000-0000-000000000000 Prim seqno : -1 First seqno : 1 Last seqno : 5 Commit cut : 5 Last vote : -1.0 Vote policy : 0 Prim JOINED : 0 State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53 Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a Name : 'mgr00' Incoming addr: '10.217.122.10:3306' Version : 6 Flags : 00 Protocols : 2 / 10 / 4 State : NON-PRIMARY Desync count : 0 Prim state : PRIMARY Prim UUID : cc9462f9-ed16-11ed-8544-d3da5d985617 Prim seqno : 5 First seqno : -1 Last seqno : 5 Commit cut : 0 Last vote : -1.0 Vote policy : 0 Prim JOINED : 1 State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53 Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a Name : 'mgr02' Incoming addr: '10.217.122.12:3306' 2023-05-07 20:47:51 0 [Warning] WSREP: No re-merged primary component found. 2023-05-07 20:47:51 0 [Warning] WSREP: No bootstrapped primary component found. 2023-05-07 20:47:51 0 [ERROR] WSREP: ./gcs/src/gcs_state_msg.cpp:gcs_state_msg_get_quorum():947: Failed to establish quorum. 2023-05-07 20:47:51 0 [Note] WSREP: Quorum results: version = 6, component = NON-PRIMARY, conf_id = -1, members = 0/3 (joined/total), act_id = -1, last_appl. = 5, protocols = -1/-1/-1 (gcs/repl/appl), vote policy= 1, group UUID = 00000000-0000-0000-0000-000000000000 2023-05-07 20:47:51 0 [Note] WSREP: Flow-control interval: [28, 28] 2023-05-07 20:47:51 0 [Note] WSREP: Received NON-PRIMARY. ``` This happens whether haproxy is enabled or not, ditto proxysql.