UnicodeEncodeError when selftest output includes non-ascii

[Gordon Tyler]

While testing Unicode handling in my mergetools branch, one of the tests failed and produced a failure report which included a unicode string (Инструмент). The selftest barfed with this UnicodeEncodeError, which does not seem to originate in my code but rather bzr's osutils.

Traceback (most recent call last):
  File "C:\dev\bzr\mergetools\bzrlib\commands.py", line 917, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "C:\dev\bzr\mergetools\bzrlib\commands.py", line 1117, in run_bzr
    ret = run(*run_argv)
  File "C:\dev\bzr\mergetools\bzrlib\commands.py", line 691, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "C:\dev\bzr\mergetools\bzrlib\commands.py", line 710, in run
    return self._operation.run_simple(*args, **kwargs)
  File "C:\dev\bzr\mergetools\bzrlib\cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "C:\dev\bzr\mergetools\bzrlib\cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "C:\dev\bzr\mergetools\bzrlib\builtins.py", line 3680, in run
    result = tests.selftest(**selftest_kwargs)
  File "C:\dev\bzr\mergetools\bzrlib\tests\__init__.py", line 3470, in selftest
    result_decorators=result_decorators,
  File "C:\dev\bzr\mergetools\bzrlib\tests\__init__.py", line 3011, in run_suite
    result = runner.run(suite)
  File "C:\dev\bzr\mergetools\bzrlib\tests\__init__.py", line 685, in run
    result.stopTestRun()
  File "C:\dev\bzr\mergetools\bzrlib\tests\__init__.py", line 489, in stopTestRun
    super(TextTestResult, self).stopTestRun()
  File "C:\dev\bzr\mergetools\bzrlib\tests\__init__.py", line 206, in stopTestRun
    self._show_list('ERROR', self.errors)
  File "build\bdist.win32\egg\testtools\testresult\real.py", line 245, in _show_list
    self.stream.write(output)
  File "C:\dev\bzr\mergetools\bzrlib\osutils.py", line 2300, in write
    data, _ = self.encode(object, self.errors)
  File "C:\Python26\lib\encodings\cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode characters in position 543-552: character maps to <undefined>

[Martin Packman (gz)]

I have an mp up fixing this blocked on updating PQM. The bug I filed isn't all that clear, and the traceback like yours is in the large first comment at the bottom of a log in the parent bug, which probably hindered launchpad finding it for you.

[Martin Pool (mbp)]

  status confirmed
  importance medium
  tags +unicode

I would guess this is specifically happening when a test fails with an
error message that includes unicode characters.

It looks like bzr thinks your output encoding is cp437. Is that right?

If we can't represent the error message in the desired output encoding
we should probably do errors=replace. In fact perhaps that should
always be true on whatever stream testtools is trying to write to?

--
Martin