testr is inconsistent with testtools over unexpected successes

Bug #1429196 reported by Chris Dent on 2015-03-06
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Testrepository
High
Unassigned

Bug Description

The following simple test file, named uxsuccess.py demonstrates that subunit does not treat unexpected success as a failure:

-=-=-

import unittest

class MyTest(unittest.TestCase):

    @unittest.expectedFailure
    def test_xfail(self):
        self.assertTrue(False)

    @unittest.expectedFailure
    def test_uxsuccess(self):
        self.assertTrue(True)
-=-=-

testtools does.

$ for i in testtools subunit ; do python -m $i.run uxsuccess.py &>/dev/null || echo "$i catches uxsuccess as fail" ; done
testtools catches uxsuccess as fail

Chris Dent (cdent) wrote :

I see that I am wrong in my initial assumption about what subunit is doing. It exits with a 0 $? whether a test suite fails or succeeds, so that means the problem is in testrepository's interpretation of the subunit stream.

testr should be detecting uxsuccess as a failure code, but it does not.

Robert Collins (lifeless) wrote :

Right, subunit exits non-zero if it fails to generate the stream; the semantics of the stream are up to the recipient. This is better for shell combinations etc where (with pipefail set, as you should) foo|bar breaks if foo exits non-zero, but bar may be filtering out some known failures.

no longer affects: subunit
Robert Collins (lifeless) wrote :

So, at the bottom of testrepository/commands/load.py:

        if not summary_result.wasSuccessful():
            return 1
        else:
            return 0

and we don't define wasSuccessful at all - thats coming directly from testtools.

Robert Collins (lifeless) wrote :

I suspect ExtendedToOriginalDecorator is triggering downlevel support somewhere.

Robert Collins (lifeless) wrote :

This is the generated stream for a modified test - with both unittest and unittest2 classes and decorators in use.

Robert Collins (lifeless) wrote :
Robert Collins (lifeless) wrote :

FWIW Upstream unittest doesn't consider unexpected success an error:

python -m unittest testrepository.tests.u
uxux
----------------------------------------------------------------------
Ran 4 tests in 0.000s

OK (expected failures=2, unexpected successes=2)

subunit2pyunit < ux.s2
testrepository.tests.u.MyTest.test_uxsuccess
testrepository.tests.u.MyTest.test_uxsuccess ... unexpected success
testrepository.tests.u.MyTest.test_xfail
testrepository.tests.u.MyTest.test_xfail ... expected failure
testrepository.tests.u.MyTest2.test_uxsuccess
testrepository.tests.u.MyTest2.test_uxsuccess ... unexpected success
testrepository.tests.u.MyTest2.test_xfail
testrepository.tests.u.MyTest2.test_xfail ... expected failure

----------------------------------------------------------------------
Ran 4 tests in 0.002s

OK (expected failures=2, unexpected successes=2)

So one arguable thing is - the downlevel issue aside - testtools is being inconsistent here.

Robert Collins (lifeless) wrote :

The stream saved by current testr - testr is still not quite v2 native (patches solicited!) so its downlevelling it to v1, but v1 has uxsuccess... not sure whats up.

e.g.
uxsuccess: testrepository.tests.u.MyTest.test_uxsuccess [ multipart
]

So at least that far into testr it is preserving the uxsuccess metadata.

Changed in testrepository:
status: New → Triaged
importance: Undecided → High
summary: - subunit does not treat unexpected success as a failure
+ testr is inconsistent with testtools over unexpected successes
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers