Record memory usage information when branch puller exceeds memory limits
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Triaged
|
Low
|
Unassigned |
Bug Description
We've recently had problems with the branch puller using way too much memory. Although we've worked around these problems by providing rlimits, it would be nice to dig at the underlying causes. Here's one way of doing that:
- Define a "soft" memory limit that is below the rlimit but still high.
- On each activity event in the puller worker, check memory usage to see that it's below that soft limit
- If it exceeds that soft limit, use py_memory_dump to get all the information we can about the current memory usage dumped to a file.
- Trigger an OOPS, terminating the worker process.
- Make sure the OOPS report includes a pointer to this file, and that the file is synced to devpad.
It's possible that some of the code written for this could be generically useful as a tool to debug extreme memory usage. If so, it should perhaps be incorporated into a hypothetical lazr.oops. However, in the meantime, let's just do this for the puller.
A simpler way would be to configure core files and examine the core file as a post mortem.