Filedescriptor leak and zombie processes

[Soren Hansen]

Something seems to be wrong with the way we run subprocesses. After running a lot of tests, everything started failing, because I had run out of file descriptors. Looking at the process table there were a bajillion zombie shells, so apparantly, we're not reaping the child processes properly. :-/

[Soren Hansen (soren)]

Turns out to be an eventlet problem.

Eventlet monkey patches os, so that wait is called with the NOHANG flag in the standard python subprocess module, but it wasn't built for that, so if the timing is right, we'll call wait before the child has terminated, and then it never gets reaped. The solution is to use eventlet's subprocess module instead, but that's broken for us: https://bitbucket.org/which_linden/eventlet/issue/77/subprocess-module-fails-if-os-module-is

[Vish Ishaya (vishvananda)]

So, is the best option to put a patched version of eventlet into the ppa and switch to using subprocess? I think this is blocking moving our staging into production.

Vish

On Feb 14, 2011, at 2:47 AM, Soren Hansen wrote:

> Turns out to be an eventlet problem.
>
> Eventlet monkey patches os, so that wait is called with the NOHANG flag
> in the standard python subprocess module, but it wasn't built for that,
> so if the timing is right, we'll call wait before the child has
> terminated, and then it never gets reaped. The solution is to use
> eventlet's subprocess module instead, but that's broken for us:
> https://bitbucket.org/which_linden/eventlet/issue/77/subprocess-module-
> fails-if-os-module-is
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/718390
>
> Title:
> Filedescriptor leak and zombie processes
>
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Something seems to be wrong with the way we run subprocesses. After
> running a lot of tests, everything started failing, because I had run
> out of file descriptors. Looking at the process table there were a
> bajillion zombie shells, so apparantly, we're not reaping the child
> processes properly. :-/
>
>

[Soren Hansen (soren)]

Yes, that's the solution.

I don't understand how it's blocking you, though. I have no reason to believe it's a new problem.

[Vish Ishaya (vishvananda)]

We are just moving to bexar. That means our current deployed code is still pre-eventlet. :) Thanks for the fix. You rock!
On Feb 14, 2011, at 1:38 PM, Soren Hansen wrote:

> Yes, that's the solution.
>
> I don't understand how it's blocking you, though. I have no reason to
> believe it's a new problem.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/718390
>
> Title:
> Filedescriptor leak and zombie processes
>
> Status in OpenStack Compute (Nova):
> In Progress
>
> Bug description:
> Something seems to be wrong with the way we run subprocesses. After
> running a lot of tests, everything started failing, because I had run
> out of file descriptors. Looking at the process table there were a
> bajillion zombie shells, so apparantly, we're not reaping the child
> processes properly. :-/
>
>