layer ordering detection is wrong

[Thomas Herve]

On recent zope.testing, some layers are detected as non-supporting teardown whereas they do (like unittest layers), thus run in a subprocess. It generally slows down the tests, and it seems generally unnecessary. From my understanding it comes from 2 things:
 * once one layer raises CanNotTearDown, all the subsequent layers are considered not supporting teardown.
 * the layer ordering is incorrect. For example the unittest layer is almost always the last one.

I guess fixing the former may be tricky, but fixing the latter looks like a sane idea.

[Thomas Herve (therve)]

I'm not sure I really understand what's going on in order_by_bases. I guess it always put the base classes before their children. It doesn't order layers between them, though.

The following patch puts the unittest layer first, but I suspect there may be a better fix:

=== modified file 'src/zope/testing/testrunner/runner.py'
--- src/zope/testing/testrunner/runner.py 2009-10-07 17:53:44 +0000
+++ src/zope/testing/testrunner/runner.py 2009-10-08 06:56:29 +0000
@@ -101,7 +101,12 @@
     def ordered_layers(self):
         layer_names = dict([(layer_from_name(layer_name), layer_name)
                             for layer_name in self.tests_by_layer_name])
- for layer in order_by_bases(layer_names):
+ from zope.testing.testrunner.layer import UnitTests
+ layers = order_by_bases(layer_names)
+ if UnitTests in layers and layers.index(UnitTests):
+ layers.remove(UnitTests)
+ layers.insert(0, UnitTests)
+ for layer in layers:
             layer_name = layer_names[layer]
             yield layer_name, layer, self.tests_by_layer_name[layer_name]

[Jonathan Lange (jml)]

FWIW, correct layer ordering requires solving the Travelling Salesman Problem. testresources has similar issues.

[Marius Gedminas (mgedmin)]

If a layer doesn't support teardown, the global state is all mucked up in the parent process, so it can no longer run any tests. This means all the following layers will be run in a fresh subprocess.

If test layers communicated the inability to be torn down in advance, rather than by raising NotImplementedError from layer.tearDown, the test runner could spawn a subprocess preemptively and keep the global state clean in the parent process.

The other thing, about layer ordering: there's a separate bug for putting the unit test layer first (bug 497871). Other than that, I don't know what "correct layer ordering" means to you. The test runner tries to reduce the number of layer.setUp/layer.tearDown calls by making sure layers that share the same base layer run next to each other, but this is not necessary for correctness, just for performance.

[Marius Gedminas (mgedmin)]

This needs to be fixed in zope.testrunner, not zope.testing.

[Colin Watson (cjwatson)]

The zope.testrunner project on Launchpad has been archived at the request of the Zope developers (see https://answers.launchpad.net/launchpad/+question/683589 and https://answers.launchpad.net/launchpad/+question/685285). If this bug is still relevant, please refile it at https://github.com/zopefoundation/zope.testrunner.