scan_branches terminated for excessive memory abuse

[Steve McInerney]

bzrsyncd 20 0 5681m 4.5g 2432 D 1 57.0 4:19.95 /usr/bin/python /srv/bzrsyncd.launchpad.net/production/launchpad/cronscripts/scan_branches.py

4.5Gb RSS, 5Gb Virt.
terminated, as that was driving us into swap.

https://pastebin.canonical.com/40916/ for a log extract.

This was while scanning ~vcs-imports/linux/btrfs

[Martin Pool (mbp)]

On 14 December 2010 16:07, Steve McInerney
<email address hidden> wrote:
> Public bug reported:
>
> bzrsyncd  20   0 5681m 4.5g 2432 D    1 57.0   4:19.95 /usr/bin/python
> /srv/bzrsyncd.launchpad.net/production/launchpad/cronscripts/scan_branches.py
>
> 4.5Gb RSS, 5Gb Virt.
> terminated, as that was driving us into swap.

Perhaps, rather than relying on it being manually terminated, we
should set a ulimit on it so that it's consistent and doesn't harm
anything else?

If we wanted to make such a change, by what technical means could we
do so (is there a branch that controls how it's run?) and who ought to
be involved in authorizing it?

--
Martin

[Martin Pool (mbp)]

After some discussion with spm and mwh:

There are really (at least) two bugs here: whatever was using the memory, and that it wasn't automatically stopped.

Other scripts set an rlimit on themselves at startup; scan_branches should too.

[Martin Pool (mbp)]

https://code.launchpad.net/~mbp/launchpad/690021-rlimit/+merge/43733 to impose an rlimit, and bug 690512 for the real underlying fix.

[Robert Collins (lifeless)]

Just happened to catch a nagios alert this afternoon: 0% swap free. AAAAAAA
Investigations following revealed:
This has probably been happening since at least Sat Apr 9 16:26:13 UTC 2011
It's happening around once or twice a day.
The process is getting to around 7Gb RSS before exhausting system memory and being killed.
It seems to be spinning somewhere to get there, is only after 7-10 minutes of run time that it achieves this.

Recent examples:
[19641951.662123] Out of memory: kill process 5894 (sh) score 2130172 or a child
[19641951.682454] Killed process 5903 (python2.6)

is for Thu May 5 04:37:19 UTC 2011
https://pastebin.canonical.com/47138/

----
[19622312.833309] Out of memory: kill process 18559 (sh) score 2333130 or a child
[19622312.873941] Killed process 18560 (python2.6)

is for Wed May 4 23:10:01 UTC 2011
https://pastebin.canonical.com/47137/

----
logs from the ps_dumper show:
https://pastebin.canonical.com/47140/
Roughly a week of processes with > 5Gb RSS.

fields are:
USER PID PPID NI PRI TIME %MEM RSS SZ VSZ STAT BLOCKED NLWP STARTED ELAPSED CMD

----
dmesg history shows:
https://pastebin.canonical.com/47139/

[Robert Collins (lifeless)]

This is effectively an OOPS (if its not killed by hand it will OOM eventually)

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12991 (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12991) by a commit, but not testable.

[Martin Pool (mbp)]

I'm unassigning myself because I do not intend to do any more on this right now, but I am happy to help with this.

My landing should make this fail cleanly/safely without hitting the kernel oomkiller. We may even get a traceback/oops out of it, which would make it easier to understand just what is going wrong. If we don't get that, it would at least be nice to identify a particular branch that blows up.

lp is currently running 2.2.3dev, and there have been a lot of memory usage improvements since then. It is possible this problem will be fixed by upgrading to bzr2.4b for which other work is underway.

[Aaron Bentley (abentley)]

Branch scan jobs now have a memory limit of 2GB. Is anything else needed to mark this fixed?

[Robert Collins (lifeless)]

Aaron says we will get an OOPS now when the job hits the memory limit and fails; so this bug is fixed - we'll get separate ones for specific branches that trigger the excessive memory use. So this bug (failures take the machine down) is fixed.