Glance-api prematurely terminates client connections

[Eugene Nikanorov]

Sometimes when glance-api workers consume lots of memory, client connections could be terminated prematurely.
When glance-api is accessed via haproxy, it results in 502 errors given to the client.

The only noticeable condition is that glance-api processes worked for half a year, and memory consumption has reached 10 Gigs per worker.

That might be that eventlet is receiving some sort of system error trying to accept incoming connection.
Failed requests never reach glance code itself, therefore they are never logged.

[Eugene Nikanorov (enikanorov)]

The only version that was discovered is 5.1

[Bug Checker Bot (bug-checker)]

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

version

expected result

steps to reproduce

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

[Kairat Kushaev (kkushaev)]

there is a workaround for that bug, it is also seems very hard to re-produce.
because of that I will mark that as medium, hope we will get some time to dig into this soon.

[Dina Belova (dbelova)]

Targeting to 5.1.1 updates.. this a medium bug, so it is not going to be fixed, but Glance team would like to check it anyway.

[Mike Fedosin (mfedosin)]

Hello! This bug was fixed during liberty cycle and backported to kilo - the idea is to introduce client socket timeout, because without it connections can exist forever: https://review.openstack.org/#/c/119132/

[Eugene Nikanorov (enikanorov)]

I think the mentioned fix is unrelated to the problem I'm describing.
Faulty glance worker did not have any connections hanging, in neither state.
What is known for sure is that glance workers consumed ~10Gig of memory, which led to memory exhaustion and could let to system error at eventlet level.

Such errors, as well as incoming requests themselves, were never logged in glance logs, so it is only a theory.

So I'd say the first thing that we need to reproduce or find corresponding bug about is memory leak.
10Gig is too much for a stateless service.

[Roman Rufanov (rrufanov)]

could it affect versions later them 5.1 ? If yes - please nominate.

[Kairat Kushaev (kkushaev)]

Invalid because the root cause was not Glance.
According to explanation from Eugene Nikanorov:

it was all qemu...
8:11 too many OSDs in their cloud. When large files are read/written, qemu opens connections to pretty mch every os, exhausting file descriptors
8:11 because the default limit is low