testr reports double failure counts

[Dan Smith]

When I run testr as part of nova's unit tests, any failures get reported twice in the count. The actual failure information is only reported once, but if two tests fail, it reports four failures at the end. This happens all the time as best I can tell.

[Russell Bryant (russellb)]

Can confirm that this happens to me as well with nova unit tests

[Michael H Wilson (geekinutah)]

I also can confirm

[Robert Collins (lifeless)]

I can't confirm this - can I see a trace of the problem please? e.g. just the output from a run where one test fails but you see two counted as failing.

[Russell Bryant (russellb)]

I can't reproduce the bug anymore, either. If it still exists, it's not as straight forward as I described it in this report. I think you can close this one. Thanks for looking!

$ printf "\n def test_fail(self):\n raise Exception()" >> nova/tests/test_utils.py
$ tox -e py27 nova.tests.test_utils
py27 develop-inst-noop: /home/rbryant/src/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | python -m nova.openstack.common.lockutils python setup.py test --slowest --testr-args=nova.tests.test_utils
running test
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --list
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpn0fbX6
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpNoCEdP
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmp9q4dRw
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpYA_RfD
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpqtYqPO
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpTftW8h
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpluy0GT
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ./nova/tests --load-list /tmp/tmpeyIcum
======================================================================
FAIL: nova.tests.test_utils.ConstantTimeCompareTestCase.test_fail
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "nova/tests/test_utils.py", line 987, in test_fail
    raise Exception()
Exception
Traceback (most recent call last):
_StringException: Empty attachments:
  pythonlogging:''
  stderr
  stdout

Traceback (most recent call last):
  File "nova/tests/test_utils.py", line 987, in test_fail
    raise Exception()
Exception

Traceback (most recent call last):
_StringException...