Buildbot: over time memory usage of the buildbot master process gets unreasonable

[Tom Haddon]

See https://pastebin.canonical.com/21521/ - buildbot is using 1331M Virtual, 427M Resident memory.

[Tom Haddon (mthaddon)]

For reference, here's what it looks like just after a restart: https://pastebin.canonical.com/21523/

[Tom Haddon (mthaddon)]

Just had another case of this. See https://pastebin.canonical.com/22438/ for details. This paste shows two buildbot instances before and after one of them was restarted (the second instance is the staging buildbot server, which was restarted a little earlier).

[Tom Haddon (mthaddon)]

And another - https://pastebin.canonical.com/23210/ - using 1G virtual 384M resident memory. Had to restart as the box was swapping.

[Gavin Panella (allenap)]

I wonder if this is related to bug #419408...

[Gavin Panella (allenap)]

To clarify what I meant in comment #4:

I described a possible circular reference problem in bug #419408, which might be impossible to garbage collect because of the presence of __del__ methods. I don't know where the circular reference might be, I'm just speculating.

However, in the code mentioned, processes not being collected would be congruent with problems during gc. Perhaps the fix for bug #419408 will help with this bug.

That's a lot of possiblys, mights, perhapses and speculation.

[Julian Edwards (julian-edwards)]

We should consider using Jenkins instead of fixing this.

[Robert Collins (lifeless)]

indeed.