assertThat style gives overly verbose output

[Martin Packman]

Currently, assertThat seems like a usability regression. To demonstrate, compare the same test from wikkid failing twice, one on a testtools subclass where assertThat is used, and one also subclassing the Bazaar case class which uses their assertEquals method rather than the testtools implementation:

Tests running...
======================================================================
FAIL: wikkid.tests.test_volatile_filestore.TestVolatileFileStore.test_mimetype
----------------------------------------------------------------------
Text attachment: traceback
------------
Traceback (most recent call last):
  File "C:\bzr\testtools\testtools\runtest.py", line 143, in _run_user
    return fn(*args)
  File "C:\bzr\testtools\testtools\testcase.py", line 539, in _run_test_method
    return self._get_test_method()()
  File "C:\bzr\wikkid\wikkid\tests\filestore.py", line 68, in test_mimetype
    self.assertEqual(
  File "C:\bzr\testtools\testtools\testcase.py", line 309, in assertEqual
    self.assertThat(observed, matcher)
  File "C:\bzr\testtools\testtools\testcase.py", line 384, in assertThat
    self.fail('Match failed. Matchee: "%s"\nMatcher: %s\nDifference: %s\n'
AssertionError: Match failed. Matchee: "None"
Matcher: Equals('text/x-c++src')
Difference: 'text/x-c++src' != None

------------
======================================================================
FAIL: wikkid.tests.test_bzr_filestore.TestBzrFileStore.test_mimetype
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\bzr\wikkid\wikkid\tests\filestore.py", line 68, in test_mimetype
    self.assertEqual(
AssertionError: not equal:
a = 'text/x-c++src'
b = None

Ran 114 tests in 5.078s

FAILED (failures=2)

Robert noted two main differences:
# The extra levels of traceback, which he finds useful as he works on the test code itself. I'd suggest most people don't, and only care about their tests. The <lp:~gz/testtools/cleaner_tracebacks> branch would fix this, but makes life harder for the Roberts of the world and there doesn't seem much interest in landing it.
# The formatting is different. Robert felt there isn't much different between them, but the assertThat version prints the expected and actual output twice and uses longer more complicated words. It should make it clearer which is the expected value however, if the test is correctly written. Also, exarkun noted he liked being able to paste the test output into a repl which the Bazaar version enables.

See also bug 660852 as a wider output verbosity issue.

[Jonathan Lange (jml)]

It's not quite comparing apples with apples, since the Equals mismatch object special-cases short inequalities.

I personally think that the default should be to hide the layers of stack that don't matter for most test authors, and that we should provide some way to make those layers more visible for test framework folk. IIRC, recent versions of unittest have some built in support for this.

[Jonathan Lange (jml)]

Stack levels are being dealt with in bug 788974.

Matchee often printed twice (once by assertThat, once by matcher as part of mismatch) is bug 593190.

Remaining issues on this ticket:
 * Matcher argument printed twice, once by assertThat as part of str(matcher), once by matcher as part of mismatch
 * "Paste test output into a repl" → not sure what this actually means or how we're failing
 * assertThat version "uses longer more complicated words" → I can kind of see this, but would like more clarity. Is this an issue with assertThat, with Equals, with both?

[Jonathan Lange (jml)]

I'm very keen to fix this bug in either this release or the next, so would appreciate input on the last two points. Stack level bug & wider verbosity bug are also being dealt with.