UnicodeDecodeError in _inaccessible_normalized_filename

[Reuben Firmin]

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 729, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 924, in run_bzr
    ret = run(*run_argv)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/commands.py", line 560, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/builtins.py", line 1143, in run
    source_branch=br_from)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/bzrdir.py", line 1195, in sprout
    hardlink=hardlink)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/bzrdir.py", line 1599, in create_workingtree
    accelerator_tree=accelerator_tree, hardlink=hardlink)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/workingtree_4.py", line 1418, in initialize
    basis = accelerator_tree.revision_tree(revision_id)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/workingtree.py", line 2054, in revision_tree
    if revision_id == self.last_revision():
  File "/usr/local/lib/python2.6/site-packages/bzrlib/workingtree.py", line 1751, in last_revision
    return self._last_revision()
  File "/usr/local/lib/python2.6/site-packages/bzrlib/decorators.py", line 136, in read_locked
    self.lock_read()
  File "/usr/local/lib/python2.6/site-packages/bzrlib/workingtree.py", line 1768, in lock_read
    self._reset_data()
  File "/root/.bazaar/plugins/git/workingtree.py", line 161, in _reset_data
    self.repository._git.object_store)
  File "/root/.bazaar/plugins/git/inventory.py", line 283, in __init__
    ie = self.add_path(path, kind, file_id, self.add_parents(path))
  File "/usr/local/lib/python2.6/site-packages/bzrlib/inventory.py", line 1288, in add_path
    ie = make_entry(kind, parts[-1], parent_id, file_id)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/inventory.py", line 2004, in make_entry
    name = ensure_normalized_name(name)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/inventory.py", line 2023, in ensure_normalized_name
    norm_name, can_access = osutils.normalized_filename(name)
  File "/usr/local/lib/python2.6/site-packages/bzrlib/osutils.py", line 1185, in _inaccessible_normalized_filename
    normalized = unicodedata.normalize('NFC', unicode(path))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)

[Martin Packman (gz)]

Can someone who has hit this give steps to reproduce? I am presuming this is related to bzr-git and non-ascii filenames, but trying some things out got bug 587074 rather than this one. I could fix the _inaccessible_normalized_filename function, but don't want to fiddle without knowing the root cause.

[Martin Packman (gz)]

It's possible that it's just the bzr-git plugin doing something wrong by passing byte strings to the bzrlib inventory code, but even if so, the bzrlib side should be much more robust.

[Martin Packman (gz)]

mkdir gitfsenc
cd gitfsenc
git init
python -c "file(u'\xe9','w').close()"
git add .
git commit -m "Initial version (filesystem encoding)"

[Martin Packman (gz)]

Missed the final step in that repo script, which was `bzr ls` though other things may also break.

The warning from bzrlib.plugins.git.inventory at the top of the log is interesting, and suggests that at the filepath handling there is part of the issue.

[Fabien Tassin (fta)]

fix released? really?

in oneiric, etckeeper is broken because of bzr:

/etc/cron.daily/etckeeper:
bzr: failed to report crash using apport:
     AssertionError()
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 34: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 936, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 1136, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 697, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.7/dist-packages/bzrlib/commands.py", line 719, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/builtins.py", line 707, in run
    no_recurse, action=action, save=not dry_run)
  File "/usr/lib/python2.7/dist-packages/bzrlib/mutabletree.py", line 49, in tree_write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/bzrlib/mutabletree.py", line 628, in smart_add
    for subf in sorted(os.listdir(abspath)):
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 34: ordinal not in range(128)

bzr 2.4b2 on python 2.7.2rc1 (Linux-2.6.39-3-generic-x86_64-with-
    Ubuntu-11.10-oneiric)
arguments: ['/usr/bin/bzr', 'add', '-q', '.']
plugins: bash_completion[2.4b2], builddeb[2.7.0dev], bzrtools[2.4.0],
    changelog_merge[2.4b2], etckeeper[unknown], launchpad[2.4b2],
    netrc_credential_store[2.4b2], news_merge[2.4b2], weave_fmt[unknown]
encoding: 'ANSI_X3.4-1968', fsenc: 'ANSI_X3.4-1968', lang: None

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.
etckeeper warning: bzr add failed

[Jelmer Vernooij (jelmer)]

That's a different bug, which is not related to bzr-git. You seem to have a non-utf8 filename in your /etc directory.

[Martin Packman (gz)]

Fabien, you want bug 715547 not this one, lots of different problems can result in a UnicodeDecodeError so where the traceback comes from matters.