buildd-queue-builder high RAM usage

[Chris Jones]

We are seeing some runs of build-queue-builder grow to virtual sizes of over 2GB and resident sizes of nearly 2GB.
This is pushing the hosting server into swap and while more RAM would prevent this, it would be good to investigate to see if the RAM usage can be reduced (especially if this is going to scale badly with an ever expanding archive).

[Celso Providelo (cprov)]

`queue-builder` load/task will be proportional to the number of sources in the DEVELOPMENT distroseries and the post-release pockets of supported distroseries. It means a big (as it was before) not a huge (as reported) amount of memory. It's quite possible that some sub-optimal approach got exacerbated by Storm adoption in this area too.

Needs investigation.

[Tom Haddon (mthaddon)]

Just saw this again:

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1000 1411 46.9 83.9 2538636 1731192 ? D 00:08 11:55 /usr/bin/python2.4 /srv/launchpad.net/codelines/current/cronscripts/buildd-queue-builder.py

Caused cesium to eat into 50% of its swap.

[Julian Edwards (julian-edwards)]

Do you get notified of this happening, or is it just when you notice it?

[Steve McInerney (spm)]

Effectively notified.
As the memory usage is quite large it spills over into swap, and we get notified of moderate swap usage.

As Chris suggested - we could add more memory and make the notifications go away. But that is probably not the best response. :-)

[Julian Edwards (julian-edwards)]

Ok thanks, we'll take a look at this in 2.1.10.

[Celso Providelo (cprov)]

I've observed the memory growing rate on the fly and it seems to be proportional with the number of published sources processed (after the initial 600MB zope offset, of course).

{{{
2008-09-26 00:08:02 INFO creating lockfile
2008-09-26 00:08:05 INFO Rebuilding Build Queue.
2008-09-26 00:08:05 INFO Buildd Master has been initialised
2008-09-26 00:08:05 INFO Processing warty
2008-09-26 00:08:05 INFO Processing hoary
2008-09-26 00:08:05 INFO Processing breezy
2008-09-26 00:08:05 INFO Processing dapper
2008-09-26 00:08:05 INFO Supported architectures: amd64 hppa i386 ia64 powerpc sparc
2008-09-26 00:08:11 INFO Found 1751 source(s) published.
2008-09-26 00:09:06 INFO Processing edgy
2008-09-26 00:09:06 INFO Supported architectures: amd64 i386 ia64 powerpc sparc
2008-09-26 00:09:09 INFO Found 391 source(s) published.
2008-09-26 00:09:15 INFO Processing feisty
2008-09-26 00:09:15 INFO Supported architectures: amd64 i386 ia64 powerpc sparc
2008-09-26 00:09:21 INFO Found 1650 source(s) published.
2008-09-26 00:10:08 INFO Processing gutsy
2008-09-26 00:10:08 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:10:13 INFO Found 3620 source(s) published.
2008-09-26 00:12:27 INFO Processing hardy
2008-09-26 00:12:27 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:12:34 INFO Found 5671 source(s) published.
2008-09-26 00:16:06 INFO Processing intrepid
2008-09-26 00:16:06 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:16:11 INFO Found 16211 source(s) published.
}}}

It looks like we can reuse the approach taken in PublishingTunableLoop and process small batches of input and clean the storm cache between batches.

There is a performance penalty doing this, but it pays off in on memory usage, keeping it stable around 800MB.