Turnip api server suffering from FD leaks

[Thomas Cuthbert]

We received pages that the git api service was returning http error 500s.

Upon closer inspection of the API logs we found the following:
https://pastebin.canonical.com/186035/

It looks like the gunicorn server is leaking FD and the system is unable to open a handle to necessary files.

Restarting the api server clears the issue.

[Colin Watson (cjwatson)]

In the past we've guessed that a libgit2 upgrade should help. I've been working on that today.

[Colin Watson (cjwatson)]

I'm obviously not working on this right now ... although I think a libgit2 upgrade is still worthwhile and the next thing to try.

[Thiago F. Pappacena (pappacena)]

It is hard to confirm that what I've found is actually the cause of this problem, but looking at the traceback and my tests to port turnip to python3, maybe the cause of this bug could be simpler than we thought.

It seems that we are not explicitly closing the config file descriptor, and it seems that we read this configuration on every request. So *maybe* we could have too many open config FDs if we have several API calls and the GC isn't running frequently enough.

I'm sending a quick MP to check if closing the FD will make this problem go away; maybe we could cache the configuration too, to avoid reading it on every single request.