assertThat(..., verbose=True) sometimes generates unprintable AssertionErrors

Bug #804127 reported by Martin Packman on 2011-07-01
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
testtools
Medium
Martin Packman

Bug Description

The assertThat method for use with matchers creates AssertionError instances that don't stringify on Python 2.4, 2.5, and some 2.6 minor versions when used with a non-ascii unicode argument. The testtools.testresults code has some robustness that means the failure is still printed, but other runners may break if the UnicodeEncodeError from trying to stringify the exception propagates.

Instead of the expected output, these older Pythons get:

Traceback (most recent call last):
  ...
    self.assertThat(u"\xa7", Equals(u"a"), verbose=True)
  File "C:\Python24\Lib\site-packages\testtools\testcase.py", line 351, in assertThat
    self.fail('Match failed. Matchee: "%s"\nMatcher: %s\nDifference: %s\n'
AssertionError: <unprintable AssertionError object>

Related branches

Jonathan Lange (jml) on 2011-07-01
tags: added: unicode
summary: - assertThat should be careful when handling unicode
+ assertThat sometimes generates unprintable AssertionErrors
Changed in testtools:
status: New → Triaged
importance: Undecided → Medium

This test seems to reproduce:

    def test_assertThat_unicode(self):
        crazy_char = _u("\xa7")
        e = self.assertRaises(
            self.failureException,
            self.assertThat, crazy_char, Equals(_u("a")), verbose=True)
        str_exception = _exception_to_text(e)
        self.assertIsNotNone(str_exception)

Also, see this discussion:

<jml> mgz: around?
<mgz> aha, I was considering pinging you
<mgz> I've got an idea that might help some of the assertThat issues
<mgz> how about just putting the Mismatch object in the AssertionError rather than stringifying it? avoids a bunch of __str__ vs __unicode__ issues
<mgz> also puts the formatting entirely in the hands of the Mismatch rather than printing the Matcher at all
<mgz> I don't think there are any bad side effects from having a non-string type argument to an exception
<mgz> jml: anything in particular?
<jml> mgz: oh huh
<jml> mgz: just wanted your feedback on a bug
<jml> mgz: https://bugs.launchpad.net/testtools/+bug/675323
<ubot5> Ubuntu bug 675323 in testtools "assertThat style gives overly verbose output" [Wishlist,Triaged]
<mgz> how would you feel about just printing the "Difference" portion and none of the rest?
<mgz> in cases like assertThat(something_long, DoctestMatches(pattern, doctest.REPORT_UDIFF)) that'll be much better anyway
<mgz> and trivial things like assertEqual(1, 2) are clear anyway without being explicit about what the Matcher and matchee are
<jml> mgz: basically, that's my plan.
<jml> mgz: maybe adding a verbose option to assertThat to print its current output
<jml> mgz: patch up.
<jml> mgz: regarding your earlier comment, you mean doing something like 'raise AssertionError(mismatch)' and relying on AssertionError to call __str__ (and thus, describe)
<mgz> I do.
<mgz> Then we can get the logic right on the Mismatch classes, and avoid the python version variations on exception instances
<jml> Hmm.
<jml> mgz: That would change the API for mismatches, not sure that would be a problem.
<mgz> In which respect?
<jml> we don't currently insist on inheriting from Mismatch
<jml> and we don't currently use mismatch.__str__
<jml> so third-party mismatches that don't inherit will break
<mgz> hm.
<jml> where 'break' means, "<testtools.matchers.Mismatch object at %x attributes=%r>" % (id(self), self.__dict__)
<mgz> the other option is raising an AssertionError subclass on some python versions.
<mgz> which is also a tricky api change.
<jml> well, we could _always_ raise an AssertionError subclass
<jml> that would only break people who were expecting an exact class
<jml> but would still go alright w/ 'except AssertionError' or isinstance() checks
<jml> mgz: in some ways, I wouldn't mind raising a specialized exception for assertThat anyway. I think it would be nice to attach the matchee & matcher objects explictly and make them programatically available
<mgz> that sounds reasonable.
<jml> hmm.
<jml> I guess the break there is that if someone sets failureException, assertThat failures become errors.
<jml> I think that's as unlikely to be a problem in practice as third-party mismatches that don't inherit from Mismatch.

description: updated
Jonathan Lange (jml) wrote :

I don't actually know how to fix this. Looking over the conversation, I'm not sure why a special exception class is called for, instead of passing in a better string.

summary: - assertThat sometimes generates unprintable AssertionErrors
+ assertThat(..., verbose=True) sometimes generates unprintable
+ AssertionErrors
Martin Packman (gz) wrote :

Certainly just mangling the string in the exception to be ascii-only on Python 2 works as well. The problem is that loses information that testtools could use.

Not having a clear interface means other matchers are likely to fall into this kind of trap with unicode, for instance the StartsWith matcher, and after <lp:~jml/testtools/nicer-regex-fail> MatchesRegex too.

Jonathan Lange (jml) wrote :

Well, yes, I agree we should avoid mangling down if we can.

I'm not clear on which bits of the interface you think are not clear.
  * Mismatch.describe() has a poorly defined return type – is it bytes or is it text?
  * Matcher objects must have a working __str__(), but I have to confess I don't know what that is supposed to return in code that works on both Python 2 & 3.
  * assertThat also calls str() on the matchee, which can be anything.
  * These three "strings" are all whacked together and passed to self.fail(), which is in turn passed to AssertionError.

Things I'm happy to change or clarify:
  * Give Mismatch.describe() a clear return type. Pick one of bytes or text. I think the latter makes sense.
  * I don't really care about Matcher.__str__. Bug 686807 shows that it's not great as is.
  * We can do whatever we want to the matchee to get a human-readable thing out of it.
  * We can raise our own exception. But if we do, it should be a subclass of AssertionError.

Whatever we do, we need to be really conscious of maintaining compatibility with Matchers outside of testtools.

Jonathan Lange (jml) on 2011-08-05
Changed in testtools:
status: Triaged → In Progress
assignee: nobody → Jonathan Lange (jml)
Jonathan Lange (jml) on 2011-09-14
Changed in testtools:
assignee: Jonathan Lange (jml) → Martin [gz] (gz)
status: In Progress → Fix Committed
Jonathan Lange (jml) on 2011-09-14
Changed in testtools:
milestone: none → next
Jonathan Lange (jml) on 2011-09-14
Changed in testtools:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers