This is really weird: from haproxy logs it's clear that nova-api doesn't respond in time, so haproxy closes a connection and returns 504 to a client. At the same time, there are no errors in nova-api logs. Moreover, there is no corresponding POST entry in nova-api log, like if the request never happened or, more precisely, never completed.
The interesting fact is that nova-api (which didn't respond in time) process has a lot of open FDs:
This is really weird: from haproxy logs it's clear that nova-api doesn't respond in time, so haproxy closes a connection and returns 504 to a client. At the same time, there are no errors in nova-api logs. Moreover, there is no corresponding POST entry in nova-api log, like if the request never happened or, more precisely, never completed.
The interesting fact is that nova-api (which didn't respond in time) process has a lot of open FDs:
root@node-200:~# lsof -p 11529 |grep "can't identify protocol" | wc -l
625
and, curiously, memcache connections:
root@node-200:~# lsof -p 11529 |grep ":11211" | wc -l
171
My current understanding is that in some rare case eventlet green threads get stuck and never finish, which results in timeout on haproxy side.