Comment 2 for bug 1427345

This is really weird: from haproxy logs it's clear that nova-api doesn't respond in time, so haproxy closes a connection and returns 504 to a client. At the same time, there are no errors in nova-api logs. Moreover, there is no corresponding POST entry in nova-api log, like if the request never happened or, more precisely, never completed.

The interesting fact is that nova-api (which didn't respond in time) process has a lot of open FDs:

  root@node-200:~# lsof -p 11529 |grep "can't identify protocol" | wc -l
  625

and, curiously, memcache connections:

  root@node-200:~# lsof -p 11529 |grep ":11211" | wc -l
  171

My current understanding is that in some rare case eventlet green threads get stuck and never finish, which results in timeout on haproxy side.