Bug #128496 “Unable to open native working tree with non-ascii f...” : Bugs : Bazaar

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2007-07-26: Re: [Bug 128496] bzr status in an svn checkout fails with SubversionException

#1

Can you perhaps provide a test case that demonstrates this bug? I at
least can't reproduce it with a simple checkout.

Revision history for this message

Wouter van Heyst (larstiq) wrote on 2007-07-26: Re: bzr status in an svn checkout fails with SubversionException

#2

That repo fails on multiple machines that handle others fine, I'll see if I can narrow it down.

Jelmer Vernooij (jelmer) on 2007-08-04

Changed in bzr-svn:
assignee:	nobody → jelmer
status:	New → Incomplete

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2007-09-09: Re: [Bug 128496] Re: bzr status in an svn checkout fails with SubversionException

#3

Ok, I can reproduce this now.

  summary "Unable to open native working tree with non-ascii filenames"
  status triaged
  importance medium

Thanks for the bug report.

--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/
Jabber: <email address hidden>

Changed in bzr-svn:
importance:	Undecided → Medium
status:	Incomplete → Triaged

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2008-01-18:

#4

Looks like svn.wc.revision_status() should be avoided because it can't deal with non-ASCII stuff.

Jelmer Vernooij (jelmer) on 2008-01-19

Changed in bzr-svn:
milestone:	none → 0.4.7

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2008-01-24:

#5

According to a thread on subversion-dev, this is not a bug but rather a problem caused by not being able to convert from the file system encoding to utf-8.

I guess this means the only thing bzr-svn can do is catch the exception and print a clear error.

Revision history for this message

Wesley J. Landaker (wjl) wrote on 2008-01-27:

#6

I think you are referring to a thread that I actually started. The conclusion that it's an encoding conversion issue is NOT true (I haven't gotten back to that thread yet to comment, due to being ill for a few days). The locale is UTF-8, and the filename is valid in UTF-8, and it's trying to convert that to UTF-8. It doesn't fail when libsvn does it, it only fails in the python bindings.

But I do believe the conclusion is correct that this is not a bzr-svn bug, but a bug somewhere in the svn python bindings.

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2008-01-28:

#7

Yes, that's the thread I'm referring to.

I don't think this is specific to the Python bindings though, as the could be reproduced with svnversion which doesn't use Python at all.

Revision history for this message

Wesley J. Landaker (wjl) wrote on 2008-01-28:

#8

Not with my reproduction recipe. The person who claimed that it could be reproduced with svnversion showed it failing on a UTF-8 name using a non-UTF-8 locale, which is supposed to fail.

If you are using a UTF-8 locale, you should be able to validate this yourself: it will work with svnversion, but fail with python svn. If you use my tarball with a non UTF-8 locale, it will always fail, as tar does not transcode filenames, and the tarball I made contains UTF-8 names. (This was the original demonstration I noted on the list. The counter example someone else posted I believe is flawed.)

Anyway, I'll take the rest of this discussion to the SVN mailing list, as at this point I am sure that this is not a bzr-svn bug. I don't know if it's useful to continue parallel discussion on this linked bug (but we can if someone else thinks it is helpful -- I just want to help get this fixed one way or another! =).

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2008-03-22:

#9

I noticed it seems like this is fixable by setting the locale appropriately from python:

import locale
locale.setlocale(locale.LC_ALL, "en_US.UTF-8")

except:

- this is valid for the complete process, so I can't just use this in bzr-svn
- I don't know what the file system encoding is

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-08: Backtrace and debugging for non-ascii filenames

#10

Download full text (5.9 KiB)

OK, I hit this as well. Did some heavy debugging. First the backtrace of where I am, mixed Python and C, the latter re-ordered to match most recent call last order.

bzr: ERROR: svn.core.SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 173, in run
    tree, file_list = tree_files(file_list)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 64, in tree_files
    return internal_tree_files(file_list, default_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 88, in internal_tree_files
    return WorkingTree.open_containing(default_branch)[0], file_list
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree.py", line 325, in open_containing
    return control.open_workingtree(), relpath
  File "~/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "~/.bazaar/plugins/svn/workingtree.py", line 88, in __init__
    status = svn.wc.revision_status(self.basedir, None, True, None, None)
  File "/usr/lib/svn-python/libsvn/wc.py", line 2310, in svn_wc_revision_status
SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

#11 0xb7725c76 in _wrap_svn_wc_revision_status ()
   from /usr/lib/python2.5/site-packages/libsvn/_wc.so
#10 0xb79083f3 in svn_wc_revision_status (result_p=0xbfe173a0,
    wc_path=0xa10e184 "/home/mvg/src/java/ornament", trail_url=0x0,
    committed=1, cancel_func=0xb7ade7fb <svn_swig_py_cancel_func>,
    cancel_baton=0x4e5168c0, pool=0xa27b108)
    at subversion/libsvn_wc/revision_status.c:123
#9 0xb7acc313 in close_edit (edit_baton=0xa30ad18, pool=0xa27b108)
    at subversion/libsvn_delta/cancel.c:334
#8 0xb790ba4b in close_edit (edit_baton=0xa30a6f0, pool=0xa27b108)
    at subversion/libsvn_wc/status.c:2033
#7 0xb79092e2 in get_dir_status (eb=0xa30a6f0, parent_entry=0x0,
    adm_access=0xa27b1d8, entry=0x0, ignore_patterns=0xa30a7a0,
    depth=svn_depth_infinity, get_all=1, no_ignore=0, skip_this_dir=0,
    status_func=0xb79080d0 <analyze_status>, status_baton=0xbfe17334,
    cancel_func=0xb7ade7fb <svn_swig_py_cancel_func>, cancel_baton=0x4e5168c0,
    pool=0xa27b108) at subversion/libsvn_wc/status.c:828
#6 0xb7976164 in svn_io_get_dirents2 (dirents=0xbfe17168,
    path=0xa27b200 "/home/mvg/src/java/ornament", pool=0xa2870a0)
    at subversion/libsvn_subr/io.c:1976
#5 0xb798263b in svn_path_cstring_to_utf8 (path_utf8=0xbfe17048,
    path_apr=0xa287840 "debugSym_4-z\303\244hlige Drehung_Tile.png",
    pool=0xa2870a0) at subversion/libsvn_subr/path.c:13...

OK, I hit this as well. Did some heavy debugging. First the backtrace of where I am, mixed Python and C, the latter re-ordered to match most recent call last order.

bzr: ERROR: svn.core.SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 173, in run
    tree, file_list = tree_files(file_list)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 64, in tree_files
    return internal_tree_files(file_list, default_branch)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 88, in internal_tree_files
    return WorkingTree.open_containing(default_branch)[0], file_list
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree.py", line 325, in open_containing
    return control.open_workingtree(), relpath
  File "~/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "~/.bazaar/plugins/svn/workingtree.py", line 88, in __init__
    status = svn.wc.revision_status(self.basedir, None, True, None, None)
  File "/usr/lib/svn-python/libsvn/wc.py", line 2310, in svn_wc_revision_status
SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

#11 0xb7725c76 in _wrap_svn_wc_revision_status ()
   from /usr/lib/python2.5/site-packages/libsvn/_wc.so
#10 0xb79083f3 in svn_wc_revision_status (result_p=0xbfe173a0, 
    wc_path=0xa10e184 "/home/mvg/src/java/ornament", trail_url=0x0, 
    committed=1, cancel_func=0xb7ade7fb <svn_swig_py_cancel_func>, 
    cancel_baton=0x4e5168c0, pool=0xa27b108)
    at subversion/libsvn_wc/revision_status.c:123
#9  0xb7acc313 in close_edit (edit_baton=0xa30ad18, pool=0xa27b108)
    at subversion/libsvn_delta/cancel.c:334
#8  0xb790ba4b in close_edit (edit_baton=0xa30a6f0, pool=0xa27b108)
    at subversion/libsvn_wc/status.c:2033
#7  0xb79092e2 in get_dir_status (eb=0xa30a6f0, parent_entry=0x0, 
    adm_access=0xa27b1d8, entry=0x0, ignore_patterns=0xa30a7a0, 
    depth=svn_depth_infinity, get_all=1, no_ignore=0, skip_this_dir=0, 
    status_func=0xb79080d0 <analyze_status>, status_baton=0xbfe17334, 
    cancel_func=0xb7ade7fb <svn_swig_py_cancel_func>, cancel_baton=0x4e5168c0, 
    pool=0xa27b108) at subversion/libsvn_wc/status.c:828
#6  0xb7976164 in svn_io_get_dirents2 (dirents=0xbfe17168, 
    path=0xa27b200 "/home/mvg/src/java/ornament", pool=0xa2870a0)
    at subversion/libsvn_subr/io.c:1976
#5  0xb798263b in svn_path_cstring_to_utf8 (path_utf8=0xbfe17048, 
    path_apr=0xa287840 "debugSym_4-z\303\244hlige Drehung_Tile.png", 
    pool=0xa2870a0) at subversion/libsvn_subr/path.c:1387
#4  0xb798f611 in svn_utf_cstring_to_utf8 (dest=0xbfe17048, 
    src=0xa287840 "debugSym_4-z\303\244hlige Drehung_Tile.png", pool=0xa2870a0)
    at subversion/libsvn_subr/utf.c:752
#3  0xb798f536 in convert_cstring (dest=0xbfe17048, 
    src=0xa287840 "debugSym_4-z\303\244hlige Drehung_Tile.png", 
    node=0xa287468, pool=0xa2870a0) at subversion/libsvn_subr/utf.c:729
#2  0xb798ed36 in convert_to_stringbuf (node=0xa287468, 
    src_data=0xa287840 "debugSym_4-z\303\244hlige Drehung_Tile.png", 
    src_length=36, dest=0xbfe16f94, pool=0xa2870a0)
    at subversion/libsvn_subr/utf.c:493
#1  0x41037672 in apr_xlate_conv_buffer () from /usr/lib/libaprutil-1.so.0
#0  iconv (cd=<value optimized out>, inbuf=<value optimized out>, inbytesleft=Could not find the frame base for "iconv".
)
    at iconv.c:36

Although the value of cd is optimized out in my lib, it can be inferred from node at frame #2 if you know the memory layout of these structures. According to utf.c from subversion, node->handle has type apr_xlate_t. This type is incomplete in apr_xlate.h from package apr-utils. Its specification is given in xlate.c. It has four pointers followed by an iconv_t member. According to iconv.h, iconv_t is a void*, but it's converted to an __gconv_t* in iconv.c, which is part of glibc. So I can use these steps to look at what iconv is actually doing:

(gdb) p (**(__gconv_t*)((void*)node->handle + 4*sizeof(void*)))
$15 = {__nsteps = 2, __steps = 0xa256f80, __data = 0xa268d38}
(gdb) p (**(__gconv_t*)((void*)node->handle + 4*sizeof(void*))).__steps[0]
$16 = {__shlib_handle = 0x0, __modname = 0x0, __counter = 1, 
  __from_name = 0xb7f06be3 "ANSI_X3.4-1968//", 
  __to_name = 0x42acde2f "INTERNAL", 
  __fct = 0x429d5db0 <__gconv_transform_ascii_internal>, 
  __btowc_fct = 0x429d47b0 <__gconv_btwoc_ascii>, __init_fct = 0, 
  __end_fct = 0, __min_needed_from = 4, __max_needed_from = 4, 
  __min_needed_to = 1, __max_needed_to = 1, __stateful = 0, __data = 0x0}
(gdb) p (**(__gconv_t*)((void*)node->handle + 4*sizeof(void*))).__steps[1]
$17 = {__shlib_handle = 0x0, __modname = 0x0, __counter = 1, 
  __from_name = 0x42acde2f "INTERNAL", 
  __to_name = 0xb7f07cc7 "ISO-10646/UTF8/", 
  __fct = 0x429d91e0 <__gconv_transform_internal_utf8>, __btowc_fct = 0, 
  __init_fct = 0, __end_fct = 0, __min_needed_from = 4, __max_needed_from = 4, 
  __min_needed_to = 1, __max_needed_to = 6, __stateful = 0, __data = 0x0}

Accoding to gconv_builtin.h, "ASCII" is an alias for "ANSI_X3.4-1968//", and the function name __gconv_transform_ascii_internal confirms this. So iconv tries to convert from ASCII to UTF-8, which is bound to fail. As my system uses UTF-8, I wonder what made it think to use ASCII as the origin charset in the first place. Needs some more debugging, but this comment here is long enough as it is, so I'll be back later.

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-08:

#11

The source locale in APR is specified using apr_os_locale_encoding which in turn uses nl_langinfo which in turn bases its result on the locale currently enabled for the application.

Two important breakpoints are "setlocale" and "apr_os_locale_encoding". With them in place from the beginnin I get this sequence:

setlocale(LC_CTYPE, NULL) // query only
setlocale(LC_CTYPE, "") // set according to environment
setlocale(LC_CTYPE, "C") // set to US-ASCII default
setlocale(LC_CTYPE, NULL)
setlocale(LC_CTYPE, "")
setlocale(LC_CTYPE, "C")
apr_os_locale_encoding // repeated

So it looks like python would not set any locale settings by default in order to provide maximum compatibility for non-locale-aware applications. bzr itself seems to have no call to setlocale. I would suggest this line somewhere in the initialization code of bzr:
locale.setlocale(locale.LC_ALL, '')

So this is neither a bug in subversion nor a bug in bzr-subversion but rather a bug in bzr itself, imo. I'll associate a branch containing a suggested fix with this bug here. It's against bzr.dev, so it won't work directly with bzr-svn, but I'd hope the bzr people merge it to bzr 1.5 as well.

Revision history for this message

Wesley J. Landaker (wjl) wrote on 2008-06-08:

#12

I can confirm that patching this into bzr fixes the problems I personally was having in relation to this bug.

import locale
locale.setlocale(locale.LC_ALL, '')

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-08:

#13

I found out that bzr contains quite a bit of code dedicated to locale issues. There is even a test suite, blackbox.test_locale, dealing with this. However, bzr seems to do things in a pythonish way, using tools provided by the python library. And those functions seem to be designed to touch the current locale setting as little as possible, but rather relies on corresponding environment settings. I can think of three possible approaches:

1. Have bzr-svn wrap all its access to libsvn in a function temporarily setting the environment according to environment settings.
+ No modifications to bzr at large
- Might need to be replicated for other applications as well
- If another application which imports bzr called setlocale, that setting gets ignored

2. Have bzr call setlocale on a global level if it is run as an application. Leave everything else alone. That's the current approach of my setlocale branch.
+ Consistent locale-aware behaviour for all plugins
+ Locale-specific formatting fo dates and so on
- I guess the python functions still use environment settings over this locale, so behaviour remains inconsistent if imported in an application that did setlocale but did not modify the environment

3. Try to make Python-specific functions honour locale as set by setlocale. I haven't tried any of this, but I guess this would require determining locale settings and adjusting the environment accordingly.
+ Consistent behaviour even when imported in another application
- Maybe less locale support when run from within a non-locale aware application
- Modifying the environment when run inside an application might be a bad idea

Any input from people with more knowledge about Python and locales highy appreciated.

Revision history for this message

Wesley J. Landaker (wjl) wrote on 2008-06-08:

#14

Download full text (5.3 KiB)

Well, almost. When adding setlocale to bzr, it does fix a lot cases, but it looks like it then exposes [an]other problem[s] in bzr-svn. For example:

# before adding setlocale
$ bzr info
bzr: ERROR: libsvn._core.SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1130, in run
    verbose=noise_level, outfile=self.outf)
  File "/usr/lib/python2.5/site-packages/bzrlib/info.py", line 315, in show_bzrdir_info
    recommend_upgrade=False)
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 88, in __init__
    status = svn.wc.revision_status(self.basedir, None, True, None, None)
  File "/var/lib/python-support/python2.5/libsvn/wc.py", line 1577, in svn_wc_revision_status
    return apply(_wc.svn_wc_revision_status, args)
SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'info']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bisect /home/wjlanda/.bazaar/plugins/bisect [1.1.0pre0]
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  cvsps /home/wjlanda/.bazaar/plugins/cvsps [unknown]
  gtk /home/wjlanda/.bazaar/plugins/gtk [0.95.0dev1]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  loom /home/wjlanda/.bazaar/plugins/loom [1.4.0dev0]
  rebase /home/wjlanda/.bazaar/plugins/rebase [0.4.0dev0]
  stats /home/wjlanda/.bazaar/plugins/stats [unknown]
  svn /home/wjlanda/.bazaar/plugins/svn [0.4.11dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

# after adding setlocale
$ bzr info
bzr: ERROR: exceptions.KeyError: 'doc/I\xc2\xb2C'

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args...

Well, almost. When adding setlocale to bzr, it does fix a lot cases, but it looks like it then exposes [an]other problem[s] in bzr-svn. For example:

# before adding setlocale
$ bzr info
bzr: ERROR: libsvn._core.SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1130, in run
    verbose=noise_level, outfile=self.outf)
  File "/usr/lib/python2.5/site-packages/bzrlib/info.py", line 315, in show_bzrdir_info
    recommend_upgrade=False)
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 88, in __init__
    status = svn.wc.revision_status(self.basedir, None, True, None, None)
  File "/var/lib/python-support/python2.5/libsvn/wc.py", line 1577, in svn_wc_revision_status
    return apply(_wc.svn_wc_revision_status, args)
SubversionException: ("Can't convert string from native encoding to 'UTF-8':", 22)

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'info']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bisect               /home/wjlanda/.bazaar/plugins/bisect [1.1.0pre0]
  bzrtools             /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  cvsps                /home/wjlanda/.bazaar/plugins/cvsps [unknown]
  gtk                  /home/wjlanda/.bazaar/plugins/gtk [0.95.0dev1]
  launchpad            /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  loom                 /home/wjlanda/.bazaar/plugins/loom [1.4.0dev0]
  rebase               /home/wjlanda/.bazaar/plugins/rebase [0.4.0dev0]
  stats                /home/wjlanda/.bazaar/plugins/stats [unknown]
  svn                  /home/wjlanda/.bazaar/plugins/svn [0.4.11dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

# after adding setlocale
$ bzr info
bzr: ERROR: exceptions.KeyError: 'doc/I\xc2\xb2C'

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1130, in run
    verbose=noise_level, outfile=self.outf)
  File "/usr/lib/python2.5/site-packages/bzrlib/info.py", line 315, in show_bzrdir_info
    recommend_upgrade=False)
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 90, in __init__
    self.base_tree = SvnBasisTree(self)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 276, in __init__
    workingtree.branch.mapping)
  File "/home/wjlanda/.bazaar/plugins/svn/repository.py", line 289, in get_fileid_map
    return self.fileid_map.get_map(self.uuid, revnum, path, mapping)
  File "/home/wjlanda/.bazaar/plugins/svn/fileids.py", line 236, in get_map
    map = self.load(revid)
  File "/home/wjlanda/.bazaar/plugins/svn/fileids.py", line 214, in load
    assert isinstance(map[urllib.unquote(filename)][0], str)
KeyError: 'doc/I\xc2\xb2C'

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'info']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bisect               /home/wjlanda/.bazaar/plugins/bisect [1.1.0pre0]
  bzrtools             /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  cvsps                /home/wjlanda/.bazaar/plugins/cvsps [unknown]
  gtk                  /home/wjlanda/.bazaar/plugins/gtk [0.95.0dev1]
  launchpad            /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  loom                 /home/wjlanda/.bazaar/plugins/loom [1.4.0dev0]
  rebase               /home/wjlanda/.bazaar/plugins/rebase [0.4.0dev0]
  stats                /home/wjlanda/.bazaar/plugins/stats [unknown]
  svn                  /home/wjlanda/.bazaar/plugins/svn [0.4.11dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

But anyway, it does fix a lot of other cases, and in general seems like a step in the right direction.

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-08:

#15

OK, with the modifications in the ~gagern/bzr-svn/bug128496 branch in addition to the setlocale in bzr I got bzr info working. I am not sure, however, that this is how things should work, and I guess there might be other places needing such adjustments as well.

Revision history for this message

Wesley J. Landaker (wjl) wrote on 2008-06-08:

#16

With the latest bug128496 branch merged, and with setlocale in place, I now get an even different error. I hope this is helpful:

$ bzr info
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xb2' in position 34: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1130, in run
    verbose=noise_level, outfile=self.outf)
  File "/usr/lib/python2.5/site-packages/bzrlib/info.py", line 315, in show_bzrdir_info
    recommend_upgrade=False)
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 90, in __init__
    self.base_tree = SvnBasisTree(self)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 348, in __init__
    add_dir_to_inv(u"", wc, None)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 338, in add_dir_to_inv
    add_dir_to_inv(subrelpath, subwc, id)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 336, in add_dir_to_inv
    False, 0, None)
  File "/var/lib/python-support/python2.5/libsvn/wc.py", line 58, in svn_wc_adm_open3
    return apply(_wc.svn_wc_adm_open3, args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb2' in position 34: ordinal not in range(128)

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'info']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bisect /home/wjlanda/.bazaar/plugins/bisect [1.1.0pre0]
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  cvsps /home/wjlanda/.bazaar/plugins/cvsps [unknown]
  gtk /home/wjlanda/.bazaar/plugins/gtk [0.95.0dev1]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  loom /home/wjlanda/.bazaar/plugins/loom [1.4.0dev0]
  rebase /home/wjlanda/.bazaar/plugins/rebase [0.4.0dev0]
  stats /home/wjlanda/.bazaar/plugins/stats [unknown]
  svn /home/wjlanda/.bazaar/plugins/svn [0.4.11dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

With the latest bug128496 branch merged, and with setlocale in place, I now get an even different error. I hope this is helpful:

$ bzr info
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xb2' in position 34: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 846, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 797, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 499, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 807, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 1130, in run
    verbose=noise_level, outfile=self.outf)
  File "/usr/lib/python2.5/site-packages/bzrlib/info.py", line 315, in show_bzrdir_info
    recommend_upgrade=False)
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 743, in open_workingtree
    return SvnWorkingTree(self, self.local_path, self.open_branch())
  File "/home/wjlanda/.bazaar/plugins/svn/workingtree.py", line 90, in __init__
    self.base_tree = SvnBasisTree(self)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 348, in __init__
    add_dir_to_inv(u"", wc, None)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 338, in add_dir_to_inv
    add_dir_to_inv(subrelpath, subwc, id)
  File "/home/wjlanda/.bazaar/plugins/svn/tree.py", line 336, in add_dir_to_inv
    False, 0, None)
  File "/var/lib/python-support/python2.5/libsvn/wc.py", line 58, in svn_wc_adm_open3
    return apply(_wc.svn_wc_adm_open3, args)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb2' in position 34: ordinal not in range(128)

bzr 1.5 on python 2.5.2 (linux2)
arguments: ['/usr/bin/bzr', 'info']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  bisect               /home/wjlanda/.bazaar/plugins/bisect [1.1.0pre0]
  bzrtools             /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.5.0]
  cvsps                /home/wjlanda/.bazaar/plugins/cvsps [unknown]
  gtk                  /home/wjlanda/.bazaar/plugins/gtk [0.95.0dev1]
  launchpad            /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  loom                 /home/wjlanda/.bazaar/plugins/loom [1.4.0dev0]
  rebase               /home/wjlanda/.bazaar/plugins/rebase [0.4.0dev0]
  stats                /home/wjlanda/.bazaar/plugins/stats [unknown]
  svn                  /home/wjlanda/.bazaar/plugins/svn [0.4.11dev0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-08:

#17

Now I also added tests to my branch, and got problems in japan and russia, the only non-latin locales available on my system.
It seems that osutils.format_date doesn't return a unicode string, but rather a byte sequence.
Fixed one instance in log.LongLogFormatter.log_revision but there might be others affected as well.
osutils.format_date(...).decode() didn't work as I would have expected.
osutils.format_date(...).decode("utf-8") did work, but I'm not sure how it would do in a non-latin1 non-utf8 locale, like some of the pre-unicode asian locales. Someone care to check?

Some possible solutions:
1. Make osutils.format_date use POSIX locale. This is current behaviour in bzr.dev but I suppose it's bad for users who don't instantly understanding english weekday names. Would reduce the worth of the whole setlocale attempt.
2. Check every occurrence of osutils.format_date and figure out whether the caller needs a unicode string, UTF-8 string or a string in current default or terminal encoding. Probably it would be best if there was a test case for each of these invocations, so one would have to figure out command line commands that use them.

I'll need some developer opinion on this before I put any more work into something that might already be doomed.

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-10: Automatic conversion between byte and unicode strings

#18

I investigated the automatic conversion between byte and unicode strings, and found a really interesting thread on the bazaar mailing list called "About encoding issues": http://thread.gmane.org/gmane.comp.version-control.bazaar-ng.general/10908

There is a function called sys.setdefaultencoding to set the encoding used for such implicit transformations. Unfortunately it usually gets removed by site.py, and should only be called before, so it's a bit tricky to use and it will affect all modules. Writing your own character encoding, it is possible to trace automatic conversions without throwing an exception each time one happens.

I realized that idea as a proof of concept in my branch https://code.launchpad.net/~gagern/bzr/str-unicode but I hope for the bzr development community to extend this further, as I can't possibly investigate all automatic character conversions in bazaar all by myself.

Specifying a regular expression using the STR_UNICODE environment variable, the output can be restricted to bzr-svn, but even there the number of automatic conversions is astronomical, and some more efficient log format is required.

Revision history for this message

Jelmer Vernooij (jelmer) wrote on 2008-06-29:

#19

Now that we control the bindings, it may actually be possible to fix this there. I wonder what the most appropriate solution here is. Wrapping all bzr-svn calls by setlocale() may be the best solution here. Unless there's some other way to tell apr the encoding of everything it receives is UTF-8...

Revision history for this message

Martin von Gagern (gagern) wrote on 2008-06-29:

#20

Jelmer Vernooij wrote:
> Wrapping all bzr-svn calls by setlocale() may be the best solution here.

Ugly, as setlocale is not guaranteed to be thread-safe, and can be quite inefficient as well.

Quoting from http://docs.python.org/lib/node745.html
"It is generally a bad idea to call setlocale() in some library routine, since as a side effect it affects the entire program. Saving and restoring it is almost as bad: it is expensive and affects other threads that happen to run before the settings have been restored."

> Unless there's some other way to tell apr the encoding of everything it
> receives is UTF-8...

I can see no alternative route through the sequence of library calls. One possibility might be some interposter library that overrides one of these functions, in order to relay it to the original implementation unless the current thread requested a fixed return value for the next call. Ugly, probably difficult to get right, and I guess none too portable either.

Two more options, both of which I would deem superior to those mentioned above:

Leave it to bzr. I have hopes to get some setlocale support merged soon: http://bundlebuggy.aaronbentley.com/request/%3C4863D8D1.405%40gmx.net%3E
I'm against ugly code in one project just to work around the deficiencies of code in another project, as this tends to make code quite unreadable. So you could try to get that fix backported in the next bzr 1.5 release, if there will be such a thing, and otherwise wait for 1.6.

Have bzr-svn modify bzrlib itself. As you can see, the fix above contains modifications to three functions, and a bit of top level code. Maybe bzr-svn can introduce this fix into bzr versions that don't provide it themselves. I don't know enough Python to be sure, but I would have thought that it should be possible to redefine these functions. Either there is a reliable way to get the source and apply the patch, or you rely on the current implementation being appropriate for all previous implementations as well, or you have a list of different implementations to choose from for different versions of bzr. The top level code could be executed during bzr-svn initialization, under the condition that the locale is still unset (i.e. set to "C"), the bzr version is known not to contain the fix, and bzrlib was actually loaded by the bzr command line tool, to stay consistent with a fix in bzr.