harvest site taking up huge amounts of RES and VIRT memory

[David Ames]

The host harvest is running on is hitting OOMs.

Apache running the harvest site is using inordinate amounts of RES and VIRT memory

Tasks: 144 total, 2 running, 142 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.5%sy, 0.0%ni, 77.2%id, 22.2%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 6100856k total, 5970344k used, 130512k free, 7732k buffers
Swap: 5861372k total, 3164400k used, 2696972k free, 54648k cached

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ SWAP COMMAND
25529 www-data 20 0 7536m 3.4g 1308 S 0 58.6 4:35.80 4.0g apache2
 8274 www-data 20 0 1991m 0 0 S 0 0.0 0:00.31 1.9g apache2
 1174 www-data 20 0 3409m 1.6g 1420 S 0 27.5 1:18.97 1.7g apache2
21863 www-data 20 0 689m 11m 528 S 0 0.2 0:12.36 678m apache2
21864 www-data 20 0 689m 12m 1644 S 0 0.2 0:11.57 676m apache2
21865 www-data 20 0 625m 0 0 S 0 0.0 0:08.01 625m apache2
21866 www-data 20 0 625m 0 0 S 0 0.0 0:08.79 625m apache2

[Nick Moffitt (nick-moffitt)]

This has caused the host it runs on to require a reboot, and interrupted my weekend during oncall rotation rather abruptly at inopportune moments.

[Nick Moffitt (nick-moffitt)]

It seems from our measurements to be a fairly recent phenomenon, within the past couple of weeks.

[Tom Haddon (mthaddon)]

We believe we've mitigated the effects by adding a robots.txt that disables crawling and a few other small changes. From what we can see, everything looks good since we made these changes (about 10 days ago).