test listing failures are not detected

[Clint Byrum]

2013-05-25 07:28:44.335 | running testr
2013-05-25 07:28:44.518 | Traceback (most recent call last):
2013-05-25 07:28:44.518 | File "/usr/lib64/python2.6/runpy.py", line 122, in _run_module_as_main
2013-05-25 07:28:44.519 | "__main__", fname, loader, pkg_name)
2013-05-25 07:28:44.519 | File "/usr/lib64/python2.6/runpy.py", line 34, in _run_code
2013-05-25 07:28:44.519 | exec code in run_globals
2013-05-25 07:28:44.520 | File "/home/jenkins/workspace/gate-python-heatclient-python26/.tox/py26/lib/python2.6/site-packages/subunit/run.py", line 120, in <module>
2013-05-25 07:28:44.520 | stdout=sys.stdout)
2013-05-25 07:28:44.521 | File "/home/jenkins/workspace/gate-python-heatclient-python26/.tox/py26/lib/python2.6/site-packages/testtools/run.py", line 175, in __init__
2013-05-25 07:28:44.522 | self.parseArgs(argv)
2013-05-25 07:28:44.522 | File "/home/jenkins/workspace/gate-python-heatclient-python26/.tox/py26/lib/python2.6/site-packages/testtools/run.py", line 214, in parseArgs
2013-05-25 07:28:44.522 | self._do_discovery(argv[2:])
2013-05-25 07:28:44.522 | File "/home/jenkins/workspace/gate-python-heatclient-python26/.tox/py26/lib/python2.6/site-packages/testtools/run.py", line 266, in _do_discovery
2013-05-25 07:28:44.523 | raise AssertionError("Unable to use discovery, must use python 2.7 "
2013-05-25 07:28:44.523 | AssertionError: Unable to use discovery, must use python 2.7 or greater, or install the discover package.

[Robert Collins (lifeless)]

I tested the basic behaviour on 2.6:
cat /tmp/t.py
raise AssertionError
python /tmp/t.py
Traceback (most recent call last):
  File "/tmp/t.py", line 1, in <module>
    raise AssertionError
AssertionError
robertc@subunit-lucid:~$ echo $?
1

so something has broken your Python 2.6 behaviour w.r.t. exception handling.

[Robert Collins (lifeless)]

In a virtualenv:
python /tmp/t.py
Traceback (most recent call last):
  File "/tmp/t.py", line 1, in <module>
    raise AssertionError
AssertionError
(subunit-2.6)robertc@subunit-lucid:~$ echo $?
1

So virtualenv per se doesn't break it.

[Robert Collins (lifeless)]

runpy looks ok for me:
python -m t
Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/tmp/t.py", line 1, in <module>
    raise AssertionError
AssertionError
(subunit-2.6)robertc@subunit-lucid:/tmp$ echo $?
255

[Robert Collins (lifeless)]

trying to reproduce within testr:

./testr run
running=false
======================================================================
FAIL: process-returncode
tags: worker-0
----------------------------------------------------------------------
Binary content:
  traceback (test/plain; charset="utf8")
Ran 1 (-333) tests
FAILED (id=917, failures=1 (-1))
(testrepository-2.6)robertc@subunit-lucid:~/personal/testrepository/working$ cat .testr.conf
[DEFAULT]
test_command=false
test_id_option=--load-list $IDFILE
test_list_option=--list
;filter_tags=worker-0

[Robert Collins (lifeless)]

the issue is that if test listing fails, no tests are scheduled to run, and the test listing codepath doesn't currently check the process exit value.

[Robert Collins (lifeless)]

With subunit v1, any output would count as a test to run and a process would be spawned; with v2, unstructed output isn't misinterpreted as tests, but exposes this bug.