v2 protocol conflates failure and error

[Colin Watson]

I've been attempting to implement the v2 protocol in a test runner, and noticed that the protocol contains no provision for distinguishing "failure" (i.e. test assertion failed) and "error" (e.g. test raised an exception) as was possible in v1. This is easily visible using the conversion tools:

  $ (echo test: foo; echo error: foo) | subunit-1to2 | subunit-2to1
  time: 2017-12-26 17:25:38.137904Z
  test: foo
  time: 2017-12-26 17:25:38.138022Z
  failure: foo [ multipart
  ]

Perhaps this is deliberate, and at any rate it's no doubt hard to change now since the protocol allocates a three-bit field for the test status which is fully-populated; but it does mean that v2 loses semantics that people might well be relying on.

Even if it can't be fixed, it would be nice to at least include some advice to implementers who might be confused by this. For example, perhaps the two cases could be disambiguated by convention at a higher level: at least for Python test output an error will normally include a traceback while a failure normally won't.