Comment 8 for bug 654474

Revision history for this message
Robert Collins (lifeless) wrote :

Ok, so you want to add another status code to the stream.

unexpectedsuccess is a combination of:
 - behaviour (the test did not raise an assertion)
 - expectation (the test was expected to raise an assertion)

We have the following status codes:
success (no exception, none expected)
failure ('assertion' exception, none expected)
error (other exception, none expected)
skip (not executed, no exception expected)
xfail (any exception, exception expected)

so adding a new status code does round that out. I remain dissatisfied with the modelling though; I would kind of like to separate the observed behaviour and the expected behaviour; a bit like jelmers patch to coerce xfail to success.

Something like:
status code becomes pure behavior:
completed
raised exception
not executed

expected behaviour becomes:
not supplied
completes
raises exception
not executed

unexpected success then is:
(outcome=completed, expected_outcome=raises exception)
xfail is:
(outcome=raises exception, expected_outcome=raises exception)

One problem with this is that xfail may be quite specific (raises /this/ exception) but subunit doesn't know enough to process that. A subject for TIP I think.