run_tests.sh broken with python-subunit-0.0.11

[Ben Nemec]

Lately when I've had to rebuild a virtual environment for testing I've had problems with run_tests.sh not working correctly. I believe I've tracked the problem down to a problem with the latest python-subunit package. When I let the venv use the latest (0.0.11) I get garbage output from the tests and it claims to have run 0 tests. However, if I pin python-subunit to 0.0.10 then everything works as expected.

The problem does not seem to affect tox, which also appears to use subunit so I'm not sure whether it's a problem with the package or run_test's use of it.

I'll attach the bad output from a test run.

Edit: Forgot to mention that if I run testr directly without the --subunit flag it seems to work fine too.

[Boris Pavlovic (boris-42)]

have the same problem.

[Ben Nemec (bnemec)]

A change to pin subunit was submitted, but needs a corresponding update to openstack-requirements: https://review.openstack.org/#/c/26654

[Robert Collins (lifeless)]

I think you have subunit 0.10 installed outside the venv or something like that. Fix that and your runs should be fixed.

[Brian Elliott (belliott)]

I do not have subunit installed outside the venv and get the same issue.

Here's a quick log of it:

http://paste.openstack.org/show/36434/

[Clark Boylan (cboylan)]

The problem is with run_tests.sh use of subunit. Specifically with the --subunit option passed to testr and tools/colorizer.py's consumption of the resulting stream. Tox does not attempt to colorize the output which is why jenkins/tox do not have the same problem. I think colorizer.py expects the old stream version only and anything it doesn't recognize is passed through to stdout which is where the extra output is coming from.

The proper fix here would be to make tools/colorizer.py aware of the new subunit format instead of pinning to an older version. The new format was created in part to make the test result streams more robust and easy to debug (we really want to be using the new version). Robert may have ideas on how to do this properly.

[Ben Nemec (bnemec)]

I've abandoned the requirements change, but I still think this needs to be fixed ASAP. It's a big time waster for existing developers who rebuild a venv and potentially blocking for anyone new to the project since they're likely to think the problem is on their end.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27311

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27311
Committed: http://github.com/openstack/nova/commit/7f262c5ae2b4d18f0198807bfdcbde207e5a4b87
Submitter: Jenkins
Branch: master

commit 7f262c5ae2b4d18f0198807bfdcbde207e5a4b87
Author: Clark Boylan <email address hidden>
Date: Mon Apr 22 19:53:25 2013 -0700

    Fix run_tests.sh usage of tools/colorizer.py.

    * run_tests.sh: tools/colorizer.py expects subunit v1 and not v2.
    Convert the subunit stream to v1 if subunit-2to1 is present (this
    indicates v2 of subunit is being used) before passing the output on to
    tools/colorizer.py.

    Fixes bug #1168516

    Change-Id: I6be917181fa8d18764db89e741190964546d0632