Using 7GBs of memory on v.l.o

[Zygmunt Krynicki]

I was looking at v.l.o on the 1st of march 2012 at around 16:34 CET.

LAVA dispatcher was using around 7GBs of memory:

Line from htop:

  PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
 8462 root 20 0 8285M 6900M 3536 S 0.0 22.8 1h38:19 /srv/lava/instances/production/bin/python2.7 /srv/lava/instances/production/bin/lava-dispatch /tmp/tmpmbAdMq --oob-fd 3

You can see the 6900M figure (RSS). I suspect that the way we handle log files might be at fault. Anyway, it probably requires deeper digging.

[Zygmunt Krynicki (zyga)]

At this time I could see the following jobs running:

13148 Running imx53-02 https://android-build.linaro.org/jenkins/job/linaro-android_imx53-ics-gcc46-freescalelt-stable-open/195/ android-build-system Feb. 21, 2012, 5:54 a.m.
13461 Running origen01 lt-origen-oneiric-20120223-0+linaro-o-ubuntu-desktop-20120223-1 lava-auto Feb. 23, 2012, 8:02 a.m.
13502 Running imx53-05 https://android-build.linaro.org/jenkins/job/linaro-android_imx53-ics-gcc46-freescalelt-stable-open/200/ android-build-system Feb. 24, 2012, 5:56 a.m.
14250 Running beaglexm03 omap3-x11-base-oneiric-20120301-0+linaro-o-developer-20120301-0 lava-auto March 1, 2012, 8:02 a.m.
14256 Running imx53-01 lt-mx5-oneiric-20120301-0+linaro-o-nano-20120301-0 lava-auto March 1, 2012, 8:02 a.m.
14257 Running imx53-03 lt-mx5-oneiric-20120301-0+linaro-o-server-20120301-0 lava-auto March 1, 2012, 8:02 a.m.
14260 Running origen02 lt-origen-oneiric-20120301-0+linaro-o-developer-20120301-0 lava-auto March 1, 2012, 8:03 a.m.
14283 Running panda09 panda-x11-base-oneiric-20120301-0+linaro-o-ubuntu-desktop-20120301-0 lava-auto March 1, 2012, 8:03 a.m.
14284 Running snowball01 lt-snowball-oneiric-20120301-0+linaro-o-alip-20120301-0 lava-auto March 1, 2012, 8:03 a.m.
14287 Running snowball03 lt-snowball-oneiric-20120301-0+linaro-o-server-20120301-0 lava-auto March 1, 2012, 8:03 a.m.
14288 Running snowball02 lt-snowball-oneiric-20120301-0+linaro-o-ubuntu-desktop-20120301-0 lava-auto March 1, 2012, 8:03 a.m.
14289 Running panda20 linaro-gfx-daily-panda-desktop-lt-x11

[Michael Hudson-Doyle (mwhudson)]

Did you look in /tmp/tmpmbAdMq? That would have given some clues.

The dispatcher loads bundles into RAM while gathering results, I guess if the log file being attached is really huge (and this can certainly happen), we might have this problem.

[Michael Hudson-Doyle (mwhudson)]

Ah, the job is still running. It has a 7.1 gig log file... & is lt-origen-oneiric-20120223-0+linaro-o-ubuntu-desktop-20120223-1

I'll kill it.

I filed bug 937443 about a similar case recently.