Getting UnicodeDecodeError on czech windows

[radekp]

This is error i am getting when trying to do basic bzr commands (e.g. bzr init):

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1
in position 58: ordinal not in range(128)
  at codecs.pyc line 178
  in write

I installed bzr-setup-0.8.2.exe from download page. My windows XP are czech localized.

[John A Meinel (jameinel)]

I believe this has already been fixed in bzr.dev

This sounds like an issue that your current path contains a non ascii character, or possibly your username.
We would need the full traceback from C:\Documents and Settings\username\.bzr.log
to let you know for sure, but bzr.dev fixed many of the encoding issues we have encountered.

[radekp (psonek2)]

Hello and thanks for reply. I am pasting requested part of bzr.log and i will try if the bug is fixed when new version is released.

this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to <email address hidden>

bzr 0.8.2 invoked on python 2.4.3.final.0 (win32)
  arguments: ['bzr']
  working dir: u'c:\\temp\\cygwin\\home\\radek\\pokus'
return code 0

this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to <email address hidden>

bzr 0.8.2 invoked on python 2.4.3.final.0 (win32)
  arguments: ['bzr', 'status']
  working dir: u'c:\\temp\\cygwin\\home\\radek\\pokus'
looking for plugins in C:\Program Files\Bazaar-NG\lib\library.zip\bzrlib\plugins
[ 3472] Mon 12:22:16.905 ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 58: ordinal not in range(128)
  at codecs.pyc line 178
  in write
Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 650, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 583, in run_bzr
  File "bzrlib\plugin.pyc", line 101, in load_plugins
  File "bzrlib\plugin.pyc", line 125, in load_from_dirs
  File "bzrlib\trace.pyc", line 106, in mutter
  File "codecs.pyc", line 501, in write
  File "codecs.pyc", line 178, in write
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 58: ordinal not in range(128)

return code 3

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Well, from the looks of it, your home directory has non ascii
characters, and when it 'mutters' into the log file, it fails because it
can't write those characters.
This should have been fixed a while ago in bzr.dev.
John
=:->
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFExg/TJdeBCYSNAAMRAmQAAKCErL2i2kUh3UzVVhwBhRjNU8CGEQCgtXOq
enJw4/lm4PwQK/236j6lLE4=
=RLSj
-----END PGP SIGNATURE-----

[radekp (psonek2)]

Hello,
with release 0.9 i am getting nearly the same error:

C:\pasky>bzr init
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1
in position 35: ordinal not in range(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyc", line 697, in run_bzr_catch_errors
  File "bzrlib\commands.pyc", line 630, in run_bzr
  File "bzrlib\plugin.pyc", line 101, in load_plugins
  File "bzrlib\plugin.pyc", line 123, in load_from_dirs
  File "bzrlib\trace.pyc", line 102, in mutter
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 35: ordinal
 not in range(128)

bzr 0.9.0 on python 2.4.3.final.0 (win32)
arguments: ['bzr', 'init']

** please send this report to <email address hidden>

[John A Meinel (jameinel)]

Thanks for the report. Your path seems okay at this point. I'm wondering what it considers your username to be.

I'm really surprised to see mutter() failing. It seems that the path is being read as a byte-string, and then it is trying to upcast to Unicode, and is failing.

I think the real bug here is that we are reading plugin directories as bytestreams.

Can you try the attached patch?

It might also be a problem with config_dir() which uses the environment variable APPDATA. And I'm not sure how windows supplied a non-ascii path in APPDATA.

[radekp (psonek2)]

Hello,
i can try the patch after weekend. Meanwhile i have found temporary solution and probably source of the problem:

C:\>echo %APPDATA%
C:\Documents and Settings\Radek Polák\Data aplikací

The problem is gone If i do:
C:\>set APPDATA=data_app

To reproduce the problem do:
C:\>set APPDATA=Data aplikací

Now if you do e.g. bzr --help, then you should get the same exception as on my box.

[John A Meinel (jameinel)]

So the question is what is the encoding of %APPDATA%?

My first guess is that it is in locale.getpreferredencoding().

I dont think it is sys.getfilesystemencoding() because that should be 'mbcs', and that wouldn't print properly in your terminal.

It could also be 'sys.stdout.encoding' because the windows terminal encoding is not necessarily the preferred encoding of files.

I have been able to recreate a bug in mutter() by doing:

>>> import bzrlib.trace
>>> bzrlib.trace('foo %s', 'bar\xb5')
^- this will raise a UnicodeDecode error while trying to format, because bar is not unicode, and has non ascii characters.

I haven't been able to come up with a satisfactory way to trap it, that doesn't raise another unicode error.

I just realized the patch I attached uses filesystemencoding() which is probably incorrect.

So we need to fix mutter() but we also need to fix load_plugins() because it won't load plugins if it can't find the directory to load them from.

[John A Meinel (jameinel)]

Re-opening this bug, because it seems to be occurring in different ways.

[John A Meinel (jameinel)]

The associated branch has fixes to mutter() so that it won't raise an exception. We may need to open a new bug if the plugin directories can't be found on Windows.