Digging further, it does appear that the initially bootstrapped master is broken somehow:
```
2023-05-07 15:06:46 0 [Warning] WSREP: Quorum: No node with complete state:
Version : 6
Flags : 0x1
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : NON-PRIMARY
Prim UUID : 00000000-0000-0000-0000-000000000000
Prim seqno : -1
First seqno : -1
Last seqno : -1
Commit cut : -1
Last vote : -1.0
Vote policy : 0
Prim JOINED : 0
State UUID : c91ea93c-ece8-11ed-bcbd-d37542ae8d5e
Group UUID : 00000000-0000-0000-0000-000000000000
Name : 'mgr02'
Incoming addr: '10.217.122.12:3306'
Version : 6
Flags : 0x2
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : SYNCED
Prim UUID : 5d5b432a-ece8-11ed-9285-be6bf8d043bf
Prim seqno : 4
First seqno : 1
Last seqno : 4
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : c91ea93c-ece8-11ed-bcbd-d37542ae8d5e
Group UUID : 78a25580-ece7-11ed-863d-5af2d3394150
Name : 'mgr00'
Incoming addr: '10.217.122.10:3306'
```
^^ is what happens after the replication error on initial bootstrap of a slave, and that slave is restarted
The donor, being restarted after the two slaves fail to come up, is aware of them but can't figure out which one is primary:
```
023-05-07 20:47:51 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6f57006e-ed18-11ed-9165-5f46e7399b53 from 1 (mgr00)
2023-05-07 20:47:51 0 [Warning] WSREP: Quorum: No node with complete state:
Version : 6
Flags : 0x1
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : PRIMARY
Prim UUID : cc9462f9-ed16-11ed-8544-d3da5d985617
Prim seqno : 5
First seqno : -1
Last seqno : 5
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53
Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a
Name : 'mgr01'
Incoming addr: '10.217.122.11:3306'
Version : 6
Flags : 00
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : NON-PRIMARY
Prim UUID : 00000000-0000-0000-0000-000000000000
Prim seqno : -1
First seqno : 1
Last seqno : 5
Commit cut : 5
Last vote : -1.0
Vote policy : 0
Prim JOINED : 0
State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53
Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a
Name : 'mgr00'
Incoming addr: '10.217.122.10:3306'
Version : 6
Flags : 00
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : PRIMARY
Prim UUID : cc9462f9-ed16-11ed-8544-d3da5d985617
Prim seqno : 5
First seqno : -1
Last seqno : 5
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : 6f57006e-ed18-11ed-9165-5f46e7399b53
Group UUID : 2d143fa4-ed15-11ed-b1b9-03a39ef3932a
Name : 'mgr02'
Incoming addr: '10.217.122.12:3306'
Digging further, it does appear that the initially bootstrapped master is broken somehow:
```
2023-05-07 15:06:46 0 [Warning] WSREP: Quorum: No node with complete state:
Version : 6 0000-0000- 0000-0000000000 00 ece8-11ed- bcbd-d37542ae8d 5e 0000-0000- 0000-0000000000 00 122.12: 3306'
Flags : 0x1
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : NON-PRIMARY
Prim UUID : 00000000-
Prim seqno : -1
First seqno : -1
Last seqno : -1
Commit cut : -1
Last vote : -1.0
Vote policy : 0
Prim JOINED : 0
State UUID : c91ea93c-
Group UUID : 00000000-
Name : 'mgr02'
Incoming addr: '10.217.
Version : 6 ece8-11ed- 9285-be6bf8d043 bf ece8-11ed- bcbd-d37542ae8d 5e ece7-11ed- 863d-5af2d33941 50 122.10: 3306'
Flags : 0x2
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : SYNCED
Prim UUID : 5d5b432a-
Prim seqno : 4
First seqno : 1
Last seqno : 4
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : c91ea93c-
Group UUID : 78a25580-
Name : 'mgr00'
Incoming addr: '10.217.
```
^^ is what happens after the replication error on initial bootstrap of a slave, and that slave is restarted
The donor, being restarted after the two slaves fail to come up, is aware of them but can't figure out which one is primary: ed18-11ed- 9165-5f46e7399b 53 from 1 (mgr00)
```
023-05-07 20:47:51 0 [Note] WSREP: STATE EXCHANGE: got state msg: 6f57006e-
2023-05-07 20:47:51 0 [Warning] WSREP: Quorum: No node with complete state:
Version : 6 ed16-11ed- 8544-d3da5d9856 17 ed18-11ed- 9165-5f46e7399b 53 ed15-11ed- b1b9-03a39ef393 2a 122.11: 3306'
Flags : 0x1
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : PRIMARY
Prim UUID : cc9462f9-
Prim seqno : 5
First seqno : -1
Last seqno : 5
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : 6f57006e-
Group UUID : 2d143fa4-
Name : 'mgr01'
Incoming addr: '10.217.
Version : 6 0000-0000- 0000-0000000000 00 ed18-11ed- 9165-5f46e7399b 53 ed15-11ed- b1b9-03a39ef393 2a 122.10: 3306'
Flags : 00
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : NON-PRIMARY
Prim UUID : 00000000-
Prim seqno : -1
First seqno : 1
Last seqno : 5
Commit cut : 5
Last vote : -1.0
Vote policy : 0
Prim JOINED : 0
State UUID : 6f57006e-
Group UUID : 2d143fa4-
Name : 'mgr00'
Incoming addr: '10.217.
Version : 6 ed16-11ed- 8544-d3da5d9856 17 ed18-11ed- 9165-5f46e7399b 53 ed15-11ed- b1b9-03a39ef393 2a 122.12: 3306'
Flags : 00
Protocols : 2 / 10 / 4
State : NON-PRIMARY
Desync count : 0
Prim state : PRIMARY
Prim UUID : cc9462f9-
Prim seqno : 5
First seqno : -1
Last seqno : 5
Commit cut : 0
Last vote : -1.0
Vote policy : 0
Prim JOINED : 1
State UUID : 6f57006e-
Group UUID : 2d143fa4-
Name : 'mgr02'
Incoming addr: '10.217.
2023-05-07 20:47:51 0 [Warning] WSREP: No re-merged primary component found. gcs_state_ msg.cpp: gcs_state_ msg_get_ quorum( ):947: Failed to establish quorum. 0000-0000- 0000-0000000000 00
2023-05-07 20:47:51 0 [Warning] WSREP: No bootstrapped primary component found.
2023-05-07 20:47:51 0 [ERROR] WSREP: ./gcs/src/
2023-05-07 20:47:51 0 [Note] WSREP: Quorum results:
version = 6,
component = NON-PRIMARY,
conf_id = -1,
members = 0/3 (joined/total),
act_id = -1,
last_appl. = 5,
protocols = -1/-1/-1 (gcs/repl/appl),
vote policy= 1,
group UUID = 00000000-
2023-05-07 20:47:51 0 [Note] WSREP: Flow-control interval: [28, 28]
2023-05-07 20:47:51 0 [Note] WSREP: Received NON-PRIMARY.
```
This happens whether haproxy is enabled or not, ditto proxysql.