Comment 10 for bug 1895248

@Michael, or rather that was false down reports due to Layer checks exceeded 10s timeouts:

Sep 23 05:53:29 standalone haproxy[12]: Server nova_osapi/standalone.ctlplane.localdomain is DOWN, reason: Layer7 timeout, check duration: 10001ms. 0 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 23 05:53:29 standalone haproxy[12]: proxy nova_osapi has no server available!
Sep 23 05:53:30 standalone haproxy[12]: 192.168.24.3:55934 [23/Sep/2021:05:52:30.108] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/60050/60050 504 398 - - ---- 54/3/2/3/0 0/0 "POST /v2.1/servers/9b195052-8294-4855-afca-f4b1591ef28c/os-interface HTTP/1.1"
[..]
Sep 23 05:53:31 standalone haproxy[12]: 192.168.24.3:55934 [23/Sep/2021:05:53:31.018] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 55/3/2/0/0 0/0 "DELETE /v2.1/servers/9b195052-8294-4855-afca-f4b1591ef28c HTTP/1.1"
[..]
Sep 23 05:53:31 standalone haproxy[12]: 192.168.24.3:56012 [23/Sep/2021:05:52:35.191] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/56195/56196 200 614 - - ---- 55/2/1/2/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 05:53:33 standalone haproxy[12]: 192.168.24.3:56012 [23/Sep/2021:05:53:33.410] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 57/2/1/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 05:53:35 standalone haproxy[12]: 192.168.24.3:56762 [23/Sep/2021:05:53:35.427] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 57/2/1/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 05:54:31 standalone haproxy[12]: 192.168.24.3:56150 [23/Sep/2021:05:52:46.248] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/105396/105396 504 398 - - ---- 57/1/0/1/0 0/0 "POST /v2.1/servers/08e1a465-e565-4194-aebf-71e6084ade4a/os-interface HTTP/1.1"
[..]
Sep 23 05:54:33 standalone haproxy[12]: 192.168.24.3:57886 [23/Sep/2021:05:54:33.303] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 59/2/0/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
Sep 23 05:54:33 standalone haproxy[12]: 192.168.24.3:56150 [23/Sep/2021:05:54:33.411] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 58/1/0/0/0 0/0 "DELETE /v2.1/servers/08e1a465-e565-4194-aebf-71e6084ade4a HTTP/1.1"
[..]
Sep 23 05:54:34 standalone haproxy[12]: Server nova_osapi/standalone.ctlplane.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 5ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
[..]
Sep 23 06:05:46 standalone haproxy[12]: Server nova_osapi/standalone.ctlplane.localdomain is DOWN, reason: Layer7 timeout, check duration: 10001ms. 0 active and 0 backup servers left. 3 sessions active, 0 requeued, 0 remaining in queue.
Sep 23 06:05:46 standalone haproxy[12]: proxy nova_osapi has no server available!
Sep 23 06:05:47 standalone haproxy[12]: 192.168.24.3:38534 [23/Sep/2021:06:04:47.236] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/60064/60064 504 398 - - ---- 58/3/2/3/0 0/0 "POST /v2.1/servers/0de8242d-8957-4250-a5c1-dc1ac3df184d/os-interface HTTP/1.1"
[..]
Sep 23 06:05:47 standalone haproxy[12]: 192.168.24.3:38534 [23/Sep/2021:06:05:47.885] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 59/3/2/0/0 0/0 "DELETE /v2.1/servers/0de8242d-8957-4250-a5c1-dc1ac3df184d HTTP/1.1"
[..]
Sep 23 06:05:48 standalone haproxy[12]: 192.168.24.3:38590 [23/Sep/2021:06:04:49.668] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/58978/58978 200 613 - - ---- 60/2/1/2/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 06:05:50 standalone haproxy[12]: 192.168.24.3:38590 [23/Sep/2021:06:05:50.879] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 61/2/1/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 06:05:52 standalone haproxy[12]: 192.168.24.3:39412 [23/Sep/2021:06:05:52.895] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 63/2/1/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
[..]
Sep 23 06:06:48 standalone haproxy[12]: 192.168.24.3:39126 [23/Sep/2021:06:05:32.400] nova_osapi nova_osapi/standalone.ctlplane.localdomain 0/0/0/76311/76311 504 398 - - ---- 56/1/0/1/0 0/0 "POST /v2.1/servers/b92df3a6-0b13-47ba-9dcd-04524abced64/os-interface HTTP/1.1"
[..]
Sep 23 06:06:49 standalone haproxy[12]: 192.168.24.3:40742 [23/Sep/2021:06:06:49.137] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/0 503 212 - - SC-- 58/2/0/0/0 0/0 "POST /v2.1/os-server-external-events HTTP/1.1"
Sep 23 06:06:49 standalone haproxy[12]: 192.168.24.3:39126 [23/Sep/2021:06:06:49.268] nova_osapi nova_osapi/<NOSRV> 0/-1/-1/-1/1 503 212 - - SC-- 57/1/0/0/0 0/0 "DELETE /v2.1/servers/b92df3a6-0b13-47ba-9dcd-04524abced64 HTTP/1.1"
Sep 23 06:06:50 standalone haproxy[12]: Server nova_osapi/standalone.ctlplane.localdomain is UP, reason: Layer7 check passed, code: 200, info: "OK", check duration: 4ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

Anyway, it seems that busy haproxy was not the cause byt side effect of serialized POST /v2.1/servers/{server}/os-interface, which is synchronous and with a single n-api worker will block everything until it returns