Support for retrying intermittently failing tests

[Jonathan Lange]

testtools should provide a way to retry intermittently failing tests, ala the https://pypi.python.org/pypi/flaky decorator that works with py.test and nose.

Ideally, we would support the @flaky decorator provided by the flaky package.

Essentially, users could specify a maximum number of attempts, N, and a minimum number of successes, M. The test would be retried up N times until there were M successes. If there are M successes, the test counts as passing. If there are not, then the test counts as failed.

Either way, we would attach the errors to the test as details, and also provide some other sort of annotation to make it clear that the test was flaky.

I think the right way to implement this is a RunTest object that wraps another arbitrary RunTest object, so that the flaky retrying logic would work with both Twisted & regular tests. This might require clarifying the interfaces provided by RunTest implementations.

[Jonathan Lange (jml)]

Would help to have a suite of interface tests for RunTest as well.

[Jonathan Lange (jml)]

Would help to have an interface for RunTest.

Would help for RunTest to have a better interface.

https://github.com/ClusterHQ/flocker/blob/master/flocker/testtools/_flaky.py implements something along these lines.