Comment 4 for bug 690021

Just happened to catch a nagios alert this afternoon: 0% swap free. AAAAAAA
Investigations following revealed:
This has probably been happening since at least Sat Apr 9 16:26:13 UTC 2011
It's happening around once or twice a day.
The process is getting to around 7Gb RSS before exhausting system memory and being killed.
It seems to be spinning somewhere to get there, is only after 7-10 minutes of run time that it achieves this.

Recent examples:
[19641951.662123] Out of memory: kill process 5894 (sh) score 2130172 or a child
[19641951.682454] Killed process 5903 (python2.6)

is for Thu May 5 04:37:19 UTC 2011
https://pastebin.canonical.com/47138/

----
[19622312.833309] Out of memory: kill process 18559 (sh) score 2333130 or a child
[19622312.873941] Killed process 18560 (python2.6)

is for Wed May 4 23:10:01 UTC 2011
https://pastebin.canonical.com/47137/

----
logs from the ps_dumper show:
https://pastebin.canonical.com/47140/
Roughly a week of processes with > 5Gb RSS.

fields are:
USER PID PPID NI PRI TIME %MEM RSS SZ VSZ STAT BLOCKED NLWP STARTED ELAPSED CMD

----
dmesg history shows:
https://pastebin.canonical.com/47139/