tox -epy27 succeeds when no tests are run

[Joe Gordon]

tox -epy27 succeeds when no tests are run.

If running 'tox -epy27 thisisabadregex'

$ tox -epy27 thisisabadregex
GLOB sdist-make: ~/nova/setup.py
py27 inst-nodeps: ~/nova/.tox/dist/nova-2013.2.a2543.g3f5ad03.zip
py27 runtests: commands[0] | python tools/patch_tox_venv.py
py27 runtests: commands[1] | python setup.py testr --slowest --testr-args=thisisabadregex
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-60} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --list
PASSED (id=385)
Slowest Tests
____________________________________________________________________________________ summary ____________________________________________________________________________________
  py27: commands succeeded
  congratulations :)

While technically correct this is misleading and I cannot imagine a case where running no tests is the expected behavior

[Matt Riedemann (mriedem)]

I've seen this too. I've ran tox with some changes to the db migration scripts for DB2 support and I had a bug which made them completely bomb out and tox never reported a problem, it just said that no tests ran. When I used run_tests.sh it correctly showed the failures.

[Sean Dague (sdague)]

We actually have some nice scripts in tempest for stuff like this.

[Matthew Treinish (treinish)]

So the tempest scripts actually don't catch this case either, but it was simple enough to fix. I pushed out:

https://review.openstack.org/#/c/121261

To catch this condition and fail if no tests are run. Once this patch and the other in progress subunit-trace patch (https://review.openstack.org/92497 ) land I'll move it over to tempest-lib and expose it as an entry point in the first release. Which should hopefully be next week assuming everything makes it through the gate reasonably quickly.

Then to fix this on the nova side you'll just need to add tempest-lib to the test-requirements and pipe the subunit stream from testr into subunit-trace to handle this.

[Davanum Srinivas (DIMS) (dims-v)]

Moving to wishlist unless some really objects :)

[Davanum Srinivas (DIMS) (dims-v)]

dims@dims-mac:~/openstack/nova$ tox -epy27 thisisabadregex
py27 create: /Users/dims/openstack/nova/.tox/py27
py27 installdeps: -r/Users/dims/openstack/nova/requirements.txt, -r/Users/dims/openstack/nova/test-requirements.txt
py27 develop-inst: /Users/dims/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh thisisabadregex
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list
Slowest Tests
The test run didn't actually run any tests
ERROR: InvocationError: '/bin/bash tools/pretty_tox.sh thisisabadregex'
________________________________________________________________________________________________________________________ summary _________________________________________________________________________________________________________________________
ERROR: py27: commands failed