Cryptic error from subunit when an import fails

[Julien Danjou]

When one of the test module fails to import for whatever reason, the raw subunit stream is output by "setup.py testr", something like:

unning testr
running=${PYTHON:-python} -m subunit.run discover ceilometer --list
Non-zero exit code (2) from test listing. stdout='\xb3)\x01@<email address hidden>test_base.TestEvaluatorBaseCl
[...]
ume.test_notifications.TestNotifications.test_volume_size_resize\x97<Y\x1e\xb3 `@W\x17text/plain;charset=utf8\rimport errors\'tests.objectstore.test_swift_middleware\xca\xc0n\x81', stderr=None
error: testr failed (3)

Which is not user friendly at all.

[Alexei Kornienko (alexei-kornienko)]

We need to patch 3 projects to fix this issue:
https://code.launchpad.net/~alexei-kornienko/subunit/bug-1271133
https://code.launchpad.net/~alexei-kornienko/testrepository/bug-1271133
https://github.com/testing-cabal/testtools/pull/77

[Alan Pevec (apevec)]

Which version of testrepository fixes this?

Does it require unittest fix http://bugs.python.org/issue19746 (resolved in Python 3.5) ?

[Jonathan Lange (jml)]

According to https://bugs.launchpad.net/testtools/+bug/1380918/comments/9, this is fully fixed.

I tried running `testr run` on testtools master, first with no modifications and got:

$ testr run
running=${PYTHON:-python} -m subunit.run testtools.tests.test_suite
Ran 1196 (+1195) tests in 2.071s
PASSED (id=0, skips=8)

Then with this patch:

$ git di
diff --git a/testtools/tests/matchers/test_basic.py b/testtools/tests/matchers/test_basic.py
index c53bc9e..2270646 100644
--- a/testtools/tests/matchers/test_basic.py
+++ b/testtools/tests/matchers/test_basic.py
@@ -1,5 +1,6 @@
 # Copyright (c) 2008-2012 testtools developers. See LICENSE for details.

+aouthaosute
 import re

 from testtools import TestCase

and got:

$ testr run
running=${PYTHON:-python} -m subunit.run testtools.tests.test_suite
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/subunit/run.py", line 149, in <module>
    main()
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/subunit/run.py", line 145, in main
    stdout=stdout, exit=False)
  File "testtools/run.py", line 171, in __init__
    self.parseArgs(argv)
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/unittest2/main.py", line 135, in parseArgs
    self.createTests()
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/unittest2/main.py", line 142, in createTests
    self.module)
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/unittest2/loader.py", line 245, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/Users/jml/.virtualenvs/testtools/lib/python2.7/site-packages/unittest2/loader.py", line 230, in loadTestsFromName
    test = obj()
  File "testtools/tests/__init__.py", line 51, in test_suite
    suites = map(lambda x: x.test_suite(), modules)
  File "testtools/tests/__init__.py", line 51, in <lambda>
    suites = map(lambda x: x.test_suite(), modules)
  File "testtools/tests/matchers/__init__.py", line 8, in test_suite
    from testtools.tests.matchers import (
  File "testtools/tests/matchers/test_basic.py", line 3, in <module>
    aouthaosute
NameError: name 'aouthaosute' is not defined
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
returncode 1
Ran 1 (-1195) tests
FAILED (id=1, failures=1 (+1))

And then with this patch:

 git di
diff --git a/testtools/tests/matchers/test_basic.py b/testtools/tests/matchers/test_basic.py
index c53bc9e..5b1304c 100644
--- a/testtools/tests/matchers/test_basic.py
+++ b/testtools/tests/matchers/test_basic.py
@@ -1,5 +1,6 @@
 # Copyright (c) 2008-2012 testtools developers. See LICENSE for details.

+import aouthaosute
 import re

 from testtools import TestCase

an...