Sounds like network issue, mysql unavailble..

haproxy traceback:

Jul  6 23:59:59 overcloud-controller-1 haproxy[7]: 10.0.0.1:58400 [06/Jul/2022:23:59:59.805] keystone_public~ keystone_public/overcloud-controller-0.internalapi.localdomain 0/0/0/28/28 200 985 - - ---- 1/1/0/0/0 0/0 "GET /v3/endpoints HTTP/1.1"
Jul  7 00:00:00 overcloud-controller-1 haproxy[7]: 10.0.0.1:58400 [06/Jul/2022:23:59:59.841] keystone_public~ keystone_public/overcloud-controller-1.internalapi.localdomain 0/0/0/308/308 500 687 - - ---- 1/1/0/0/0 0/0 "POST /v3/endpoints HTTP/1.1"
Jul  7 00:03:01 overcloud-controller-1 haproxy[7]: Backup Server mysql/overcloud-controller-2.internalapi.localdomain is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 12ms. 0 active and 2 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul  7 00:03:13 overcloud-controller-1 haproxy[7]: Backup Server mysql/overcloud-controller-2.internalapi.localdomain is UP, reason: Layer7 check passed, code: 200, check duration: 14ms. 0 active and 3 backup servers online. Running on backup. 0 sessions requeued, 0 total in queue.
Jul  7 00:03:46 overcloud-controller-1 haproxy[7]: Backup Server mysql/overcloud-controller-0.internalapi.localdomain is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 10ms. 0 active and 2 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul  7 00:03:47 overcloud-controller-1 haproxy[7]: Backup Server mysql/overcloud-controller-2.internalapi.localdomain is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 17ms. 0 active and 1 backup servers left. Running on backup. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul  7 00:03:47 overcloud-controller-1 haproxy[7]: Backup Server mysql/overcloud-controller-1.internalapi.localdomain is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 11ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Jul  7 00:03:47 overcloud-controller-1 haproxy[7]: proxy mysql has no server available!


mariadb traceback:


2022-07-06 23:59:13 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection to peer e6d87958-a9ec with addr tcp://172.17.0.69:4567 timed out, no messages seen in PT3S, socket stats: rtt: 5364 rttvar: 9533 rto: 1648000 lost: 1 last_data_recv: 3387 cwnd: 1 last_queued_since: 499910544 last_delivered_since: 3387023112 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
2022-07-06 23:59:13 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') turning message relay requesting on, nonlive peers: tcp://172.17.0.69:4567 
2022-07-06 23:59:13 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection to peer e39ccd47-8f20 with addr tcp://172.17.0.78:4567 timed out, no messages seen in PT3S, socket stats: rtt: 2058 rttvar: 3323 rto: 203000 lost: 0 last_data_recv: 3445 cwnd: 10 last_queued_since: 443915138 last_delivered_since: 3444051673 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
2022-07-06 23:59:14 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') reconnecting to e6d87958-a9ec (tcp://172.17.0.69:4567), attempt 0
2022-07-06 23:59:14 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection established to e6d87958-a9ec tcp://172.17.0.69:4567
2022-07-06 23:59:14 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') reconnecting to e39ccd47-8f20 (tcp://172.17.0.78:4567), attempt 0
2022-07-06 23:59:14 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection established to e39ccd47-8f20 tcp://172.17.0.78:4567
2022-07-06 23:59:18 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') turning message relay requesting off
2022-07-06 23:59:57 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection to peer e6d87958-a9ec with addr tcp://172.17.0.69:4567 timed out, no messages seen in PT3S, socket stats: rtt: 6834 rttvar: 12311 rto: 1656000 lost: 1 last_data_recv: 3445 cwnd: 1 last_queued_since: 445522887 last_delivered_since: 3445158510 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
2022-07-06 23:59:57 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') turning message relay requesting on, nonlive peers: tcp://172.17.0.69:4567 
2022-07-06 23:59:58 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection to peer e39ccd47-8f20 with addr tcp://172.17.0.78:4567 timed out, no messages seen in PT3S, socket stats: rtt: 4535 rttvar: 8389 rto: 1640000 lost: 1 last_data_recv: 3052 cwnd: 1 last_queued_since: 499889193 last_delivered_since: 3051870157 send_queue_length: 0 send_queue_bytes: 0 segment: 0 messages: 0
2022-07-06 23:59:58 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') reconnecting to e6d87958-a9ec (tcp://172.17.0.69:4567), attempt 0
2022-07-06 23:59:59 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') reconnecting to e39ccd47-8f20 (tcp://172.17.0.78:4567), attempt 0
2022-07-06 23:59:59 0 [Note] WSREP: evs::proto(efe49719-b3f0, OPERATIONAL, view_id(REG,e39ccd47-8f20,17)) suspecting node: e6d87958-a9ec
2022-07-06 23:59:59 0 [Note] WSREP: evs::proto(efe49719-b3f0, OPERATIONAL, view_id(REG,e39ccd47-8f20,17)) suspected node without join message, declaring inactive
2022-07-06 23:59:59 0 [Note] WSREP: (efe49719-b3f0, 'tcp://172.17.0.68:4567') connection established to e6d87958-a9ec tcp://172.17.0.69:4567