testr is inconsistent with testtools over unexpected successes

[Chris Dent]

The following simple test file, named uxsuccess.py demonstrates that subunit does not treat unexpected success as a failure:

-=-=-

import unittest

class MyTest(unittest.TestCase):

    @unittest.expectedFailure
    def test_xfail(self):
        self.assertTrue(False)

    @unittest.expectedFailure
    def test_uxsuccess(self):
        self.assertTrue(True)
-=-=-

testtools does.

$ for i in testtools subunit ; do python -m $i.run uxsuccess.py &>/dev/null || echo "$i catches uxsuccess as fail" ; done
testtools catches uxsuccess as fail

[Chris Dent (cdent)]

I see that I am wrong in my initial assumption about what subunit is doing. It exits with a 0 $? whether a test suite fails or succeeds, so that means the problem is in testrepository's interpretation of the subunit stream.

testr should be detecting uxsuccess as a failure code, but it does not.

[Robert Collins (lifeless)]

Right, subunit exits non-zero if it fails to generate the stream; the semantics of the stream are up to the recipient. This is better for shell combinations etc where (with pipefail set, as you should) foo|bar breaks if foo exits non-zero, but bar may be filtering out some known failures.

[Robert Collins (lifeless)]

So, at the bottom of testrepository/commands/load.py:

        if not summary_result.wasSuccessful():
            return 1
        else:
            return 0

and we don't define wasSuccessful at all - thats coming directly from testtools.

[Robert Collins (lifeless)]

I suspect ExtendedToOriginalDecorator is triggering downlevel support somewhere.

[Robert Collins (lifeless)]

This is the generated stream for a modified test - with both unittest and unittest2 classes and decorators in use.

[Robert Collins (lifeless)]

FWIW Upstream unittest doesn't consider unexpected success an error:

python -m unittest testrepository.tests.u
uxux
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK (expected failures=2, unexpected successes=2)

subunit2pyunit < ux.s2
testrepository.tests.u.MyTest.test_uxsuccess
testrepository.tests.u.MyTest.test_uxsuccess ... unexpected success
testrepository.tests.u.MyTest.test_xfail
testrepository.tests.u.MyTest.test_xfail ... expected failure
testrepository.tests.u.MyTest2.test_uxsuccess
testrepository.tests.u.MyTest2.test_uxsuccess ... unexpected success
testrepository.tests.u.MyTest2.test_xfail
testrepository.tests.u.MyTest2.test_xfail ... expected failure

----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK (expected failures=2, unexpected successes=2)

So one arguable thing is - the downlevel issue aside - testtools is being inconsistent here.

[Robert Collins (lifeless)]

The stream saved by current testr - testr is still not quite v2 native (patches solicited!) so its downlevelling it to v1, but v1 has uxsuccess... not sure whats up.

e.g.
uxsuccess: testrepository.tests.u.MyTest.test_uxsuccess [ multipart
]

So at least that far into testr it is preserving the uxsuccess metadata.