testtools.run --list-tests exits 0 when discover fails to import

[Matthew Treinish]

If there is an error or an issue with a test file when testtools.run --list-tests is run that would cause the file to fail on import then the command still exits with 0 despite the import failure. The test list just contains an entry with the test_id being:

unittest.loader.ModuleImportFailure.$(test filename)

for each file that fails to be imported

[Robert Collins (lifeless)]

So this breaks testrepository quite badly, because it does (pseudo code):

tests = --list-tests
filter tests by user pattern
run selected tests

And if the users pattern doesn't match unittest.loader.ModuleImportFailure.$(test filename) (e.g. they listed the parent module) then their tests aren't run, but some tests may be, and the loader synthetic errors are not generated either. So there's no heuristic today that can detect this and correct it.

I think testtools can do something here - e.g. list-tests should exit 1 if a module failed to import.

[Robert Collins (lifeless)]

http://bugs.python.org/issue19746