Comment 6 for bug 259461

I've observed the memory growing rate on the fly and it seems to be proportional with the number of published sources processed (after the initial 600MB zope offset, of course).

{{{
2008-09-26 00:08:02 INFO creating lockfile
2008-09-26 00:08:05 INFO Rebuilding Build Queue.
2008-09-26 00:08:05 INFO Buildd Master has been initialised
2008-09-26 00:08:05 INFO Processing warty
2008-09-26 00:08:05 INFO Processing hoary
2008-09-26 00:08:05 INFO Processing breezy
2008-09-26 00:08:05 INFO Processing dapper
2008-09-26 00:08:05 INFO Supported architectures: amd64 hppa i386 ia64 powerpc sparc
2008-09-26 00:08:11 INFO Found 1751 source(s) published.
2008-09-26 00:09:06 INFO Processing edgy
2008-09-26 00:09:06 INFO Supported architectures: amd64 i386 ia64 powerpc sparc
2008-09-26 00:09:09 INFO Found 391 source(s) published.
2008-09-26 00:09:15 INFO Processing feisty
2008-09-26 00:09:15 INFO Supported architectures: amd64 i386 ia64 powerpc sparc
2008-09-26 00:09:21 INFO Found 1650 source(s) published.
2008-09-26 00:10:08 INFO Processing gutsy
2008-09-26 00:10:08 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:10:13 INFO Found 3620 source(s) published.
2008-09-26 00:12:27 INFO Processing hardy
2008-09-26 00:12:27 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:12:34 INFO Found 5671 source(s) published.
2008-09-26 00:16:06 INFO Processing intrepid
2008-09-26 00:16:06 INFO Supported architectures: amd64 hppa i386 ia64 lpia powerpc sparc
2008-09-26 00:16:11 INFO Found 16211 source(s) published.
}}}

It looks like we can reuse the approach taken in PublishingTunableLoop and process small batches of input and clean the storm cache between batches.

There is a performance penalty doing this, but it pays off in on memory usage, keeping it stable around 800MB.