Cannot rerun failing when the failure is an import one

[Jonathan Lange]

I'm using testr to run some tests where I'm moving things around. There are module import failures.

jml@truth:~/src/pkgme-binary/trunk$ testr run
running=PYTHONPATH=`pwd`:$PYTHONPATH python -m subunit.run discover devportalbinary
======================================================================
ERROR: unittest.loader.ModuleImportFailure.tests.test_binary_backend
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_binary_backend
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/home/jml/src/pkgme-binary/trunk/devportalbinary/tests/test_binary_backend.py", line 8, in <module>
    from pkgme.backend import ExternalHelpersBackend, get_backend_dir
ImportError: No module named pkgme.backend
======================================================================
ERROR: unittest.loader.ModuleImportFailure.tests.test_binary
----------------------------------------------------------------------
ImportError: Failed to import test module: tests.test_binary
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 252, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
    __import__(name)
  File "/home/jml/src/pkgme-binary/trunk/devportalbinary/tests/test_binary.py", line 7, in <module>
    from devportalbinary.binary import (
  File "devportalbinary/binary.py", line 41, in <module>
    from pkgme.info_elements import (
ImportError: No module named pkgme.info_elements
id=14, tests=2, failures=2

When I re-run the tests with 'run --failing', I get this:

jml@truth:~/src/pkgme-binary/trunk$ testr run --failing
running=PYTHONPATH=`pwd`:$PYTHONPATH python -m subunit.run discover unittest.loader.ModuleImportFailure.tests.test_binary unittest.loader.ModuleImportFailure.tests.test_binary_backend
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.7/dist-packages/subunit/run.py", line 73, in <module>
    stdout=sys.stdout)
  File "/usr/lib/python2.7/dist-packages/testtools/run.py", line 155, in __init__
    self.parseArgs(argv)
  File "/usr/lib/python2.7/dist-packages/testtools/run.py", line 194, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib/python2.7/dist-packages/testtools/run.py", line 303, in _do_discovery
    self.test = loader.discover(start_dir, pattern, top_level_dir)
  File "/usr/lib/python2.7/unittest/loader.py", line 202, in discover
    raise ImportError('Start directory is not importable: %r' % start_dir)
ImportError: Start directory is not importable: 'unittest.loader.ModuleImportFailure.tests.test_binary'
id=15, tests=0

Note that the test ids are incorrect: unittest.loader.ModuleImportFailure.tests.test_binary unittest.loader.ModuleImportFailure.tests.test_binary_backend rather than devportalbinary.tests.test_binary devportalbinary.tests.test_binary_backend.

[Robert Collins (lifeless)]

Testrepository doesn't know what an 'import failure' is; this is up to the runner to report test ids that can be fed back in to load-list or similar in a meaningful way.

[Jonathan Lange (jml)]

OK. Reassigning to subunit, since that's the runner here.

[Robert Collins (lifeless)]

Subunit doesn't implement a runner, it only implements a reporter for a runner - the one in testtools.

[Robert Collins (lifeless)]

I will note that perhaps testrepository needs some knowledge/signalling to enable this as well.