Use testresources for bzr selftest

[Martin Packman]

The Bazaar testsuite should have lp:testresources integrated so tests can depend on expensive or hard to manage resources without needing to setUp/tearDown them for every test. This should allow some test servers to be moved out of process without becoming a giant performance hit, and also fix other niggles like bug 421053 that involve creating or cleaning up resources,

As a new dependency, this will mean PQM will need it installed, build dependencies and packages need updating, and documentation writing.

[Martin Packman (gz)]

So, I've had a closer look at this, lp:testresources basically consists of:

* A protocol for setting up, resetting, and tearing down resources for tests.
* Fancy graph code for reordering tests to put those with similar resources adjacent.

Confusingly Robert also has a newer project, lp:python-fixtures that also covers the first case, and is a build-time dependency of testresources. Bothering him about what he thinks is the best way of adding resource reusability to the bzr test suite therefore seems like a good idea.

There are going to be a few tricky things in implementing this. Firstly, Bazaar just has a lot of tests. We don't want to touch too many of them, as that always risks nullifying the assertion. This means a lot of tests are still going to be setting up their own branches from scratch each time. The overlap with existing tooling will be interesting, Scenarios and Features have a similar general pattern for different purposes. Specifying a test using more than one of these may get messy. For instance, if I have a series of tests that need a temporary directory with a unicode name, how are UnicodeFilenameFeature and say, TempDirResource used? Are both warranted? How would scenarios using with different resource needs be set up?

The suite reordering for resource reusage needs to be integrated with the existing reordering done by selftest. Randomisation isn't a big problem, it's not widely used and could just disable the other options. However, spliting tests between children currently takes one-for-me, one-for-you approach that evens out the total times taken but splits up sibling tests. This would either need changing or the split would have to happen first.

[Martin Pool (mbp)]

Thinking about this a bit more:

Using testresources is not an end in itself, just a means to run
long-lived servers cleanly; if there's is not a good cost benefit we
can do something else. In some ways testresources is overkill for
this: the server does not have ordering constraints or conflicts and
it cannot be cleaned or reset: we just need to start it when it's
first needed (per process) and then later kill it.

There are already other long-lived resources, such as the working directory.

We _could_ just create an out-of-process smart server which is held by
a process-global and created when first used, and then cleaned up by
teardown of the test suite or atexit of the whole process.

[Martin Packman (gz)]

I've asked Robert about this, and he's of the opinion Bazaar needs both parts of the functionality, and should use fixtures for the protocol, and testresources for the ordering and general management. He's got some plans for superseding that second part with fixtures as well at some point, but at the moment they basically can't do what's required on their own due to lack of introspectability.