testtools.run --list-tests exits 0 when discover fails to import

Bug #1245672 reported by Matthew Treinish on 2013-10-28
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
subunit
Critical
Robert Collins
testtools
Critical
Robert Collins

Bug Description

If there is an error or an issue with a test file when testtools.run --list-tests is run that would cause the file to fail on import then the command still exits with 0 despite the import failure. The test list just contains an entry with the test_id being:

unittest.loader.ModuleImportFailure.$(test filename)

for each file that fails to be imported

Related branches

Changed in testtools:
status: New → Triaged
importance: Undecided → Critical
Robert Collins (lifeless) wrote :

So this breaks testrepository quite badly, because it does (pseudo code):

tests = --list-tests
filter tests by user pattern
run selected tests

And if the users pattern doesn't match unittest.loader.ModuleImportFailure.$(test filename) (e.g. they listed the parent module) then their tests aren't run, but some tests may be, and the loader synthetic errors are not generated either. So there's no heuristic today that can detect this and correct it.

I think testtools can do something here - e.g. list-tests should exit 1 if a module failed to import.

Changed in subunit:
status: New → Triaged
importance: Undecided → Critical
Changed in testtools:
status: Triaged → Fix Committed
Changed in testtools:
milestone: none → next
Changed in testtools:
status: Fix Committed → Fix Released
Changed in subunit:
status: Triaged → Fix Committed
assignee: nobody → Robert Collins (lifeless)
Changed in testtools:
assignee: nobody → Robert Collins (lifeless)
Changed in subunit:
milestone: none → next
Changed in subunit:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.