twistd executive can be paged out by memory-heavy builds leading to timeouts and the build being killed

[Julian Edwards]

If there's a memory-heavy build going on like the kernel, there's a high chance that the manager will get swapped out. This can cause undue delays on the manager side when it's trying to poll the slave.

mlockall() prevents paging out but I don't know how to do this in Python.

[Robert Collins (lifeless)]

> mlockall() prevents paging out but I don't know how to do this in
> Python.

ctypes

(read up on that, it will be straight forward.

-Rob

[Julian Edwards (julian-edwards)]

See also bug 586359 which talks about the timeout problems.

[Julian Edwards (julian-edwards)]

This works:
{{{
In [1]: from ctypes import cdll
In [2]: libc = cdll.LoadLibrary("libc.so.6")
In [3]: libc.mlockall(3)
Out[3]: 0
}}}

However it requires the CAP_IPC_LOCK capability.

[Robert Collins (lifeless)]

So the patch is straight forward, just need to ensure the script will run with sufficient perms, which is an RT level issue.

[Julian Edwards (julian-edwards)]

On Wed Jan 11 20:12:57 2012, Robert Collins wrote:
> ** Summary changed:
>
> - The twistd executive manager process should never get paged out
> + twistd executive can be paged out by memory-heavy builds leading to timeouts and the build being killed
>

This is getting more important to fix. To work around the problem that
exists mostly on arm buildds the buildd-manager's (global) timeout is
now set to be days. This is suboptimal when we need to clear up stuck
buildds of other architectures!

[Julian Edwards (julian-edwards)]

Bumping to critical due to increased problems lately, mostly caused by the 2-day timeout configured on buildd-manager. Webops are getting frustrated.

[Adam Conrad (adconrad)]

While this bug/misfeature is still around, its impact in production was mitigated last year via kernel tweaks on the ARM buildds. We're about to get a whole new set of ARM builders with different kernels, and this may or may not crop back up. If it does, I'll see what I can do to make it go away but I'm inclined right now to drop the priority of this bug.