Comment 21 for bug 605543

I tried Foppe's suggestion, but now I get this error:

Gwibber Dispatcher: ERROR <twitter:receive> Operation failed
Gwibber Dispatcher: DEBUG Checking message 20614540018 timestamp (2010-08-08 10:16:04.00) to see if it is newer than 2010-08-08 13:11:59.38
Gwibber Dispatcher: DEBUG Traceback:
Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/gwibber/microblog/dispatcher.py", line 88, in perform_operation
    m["rtl"] = util.isRTL(re.sub(text_cleaner, "", m["text"].decode('utf-8')))
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 32-34: invalid data

I think what's happening is that unescape() is now parsing a UTF-8 encoded 8-bit string, but the entity replacements it's appending to the result are actually latin-1 encoded. So the result is a mix of UTF-8 and latin-1 encoded characters.