TestCase instances may not be collected till selftest finishes

[Martin Packman]

<https://lists.ubuntu.com/archives/bazaar/2009q4/065581.html>
>> The old TestCase.run had this finally code, is there an equivalent in testtools?
>>
>> saved_attrs = {}
>> for attr_name in self.attrs_to_keep:
>> if attr_name in self.__dict__:
>> saved_attrs[attr_name] = self.__dict__[attr_name]
>> self.__dict__ = saved_attrs
>
> That code is replaced by discarding tests that have been run, a much
> simpler way of freeing memory.

This turns out to not actually be true in a number of importance cases. The upshot of which is up to 200% higher peak memory usage, with associated longer runtime, and symptoms like bug 392127 and other OOM related test failures.

<https://lists.ubuntu.com/archives/bazaar/2010q2/068602.html>
Has some specific measurements on memory usage.

So, why aren't the TestCase instances collected after being run, as planned?
* There's cycle created by testtools between the TestCase and the ContentType containing the log
* In Python 2.7 there are bound methods assert<Type>Equal stored on the instance _type_equality_funcs attribute creating cycles
* Likely other test-specific cycles especially with the existing resource leaks
* TestDecorator classes with __iter__ methods keep test list copies
* ...more?

The cycles at least will get GCed, but evidently there are enough things surviving till the run completes to waste hundreds of megabytes of memory and cause spurious failures on my box.

Given that even if this is fixed, it will be hard not to accidentally introduce regressions by leaving some references lying around, I wonder if just reinstating something similar to the old hack wouldn't be easiest.

[Martin Packman (gz)]

Unfortunately I don't think fixing this will help babune on windows (even though it helps me) as running the test suite in small pieces should mitigate this issue anyway.

[Robert Collins (lifeless)]

Ugh.

So we can reintroduce the hack, but I'd deeply love to just fix the
underlying problems here, or so something that won't just be a mess /
only apply to some tests / have gotchas and surprises like the other
thing did.

[Martin Packman (gz)]

Okay, I can have a stab at this from two directions.

Just clearing the TestCase __dict__ in a TestResult.stopTest method does work, and all it breaks are some selftest tests that want to interrogate the case after running it.

To instead ensure that TestCase instances do expire, I'd want to break all cycles manually in stopTest, and then if the test is then still alive at some future point, complain very loudly. That'll then mean changing most of the TestDecorator classes and probably some specific tests to then shut up the whining.

[John A Meinel (jameinel)]

Confirming this exists, though maybe the actual bug description is a bit off? Isn't this more "TestCases may leak memory while running the test suite?"

[Martin Packman (gz)]

Fixing this is becoming increasingly necessary to get a clean run of the full test suite on a number of platforms, and apparently on pypy as well. I'm going to try and get the remaining parts of my branch landed this week,