commit fails if the working tree contains an unknown file with undecodable name

[Lukáš Lalinský]

I was testing something in my application and I needed to added a latin-2 encode file name to the working tree, which is according to my locale settings should be utf-8. When committing to bzr, I get this error:

$ bzr ci
Committing to: /home/lukas/projects/picard/main/
modified picard/tagger.py
bzr: ERROR: exceptions.UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid data

Traceback (most recent call last):
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 806, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 762, in run_bzr
    ret = run(*run_argv)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/builtins.py", line 2328, in run
    author=author)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/workingtree_4.py", line 246, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/decorators.py", line 165, in write_locked
    return unbound(self, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/mutabletree.py", line 187, in commit
    revprops=revprops, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commit.py", line 367, in commit
    message = message_callback(self)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/builtins.py", line 2308, in get_message
    output_encoding=bzrlib.user_encoding)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/msgeditor.py", line 259, in make_commit_message_template_encoded
    template = make_commit_message_template(working_tree, specific_files)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/msgeditor.py", line 241, in make_commit_message_template
    to_file=status_tmp)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/status.py", line 114, in show_tree_status
    want_unversioned=want_unversioned)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/tree.py", line 93, in changes_from
    want_unversioned=want_unversioned,
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/tree.py", line 707, in compare
    want_unversioned=want_unversioned)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/delta.py", line 217, in _compare_trees
    want_unversioned=want_unversioned):
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/workingtree_4.py", line 2381, in _iter_changes
    (None, utf8_decode(current_path_info[0])[0]),
  File "encodings/utf_8.py", line 16, in decode
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0-2: invalid data

Since the file in question is not versioned and 'bzr ci' shouldn't really care about unknown files, I think it would be better to just skip it.

Similar problem happens with 'bzr qci':

Traceback (most recent call last):
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 806, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 762, in run_bzr
    ret = run(*run_argv)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/lukas/.bazaar/plugins/qbzr/__init__.py", line 166, in run
    window = CommitWindow(tree, selected_list)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/lazy_import.py", line 125, in __call__
    return obj(*args, **kwargs)
  File "/home/lukas/.bazaar/plugins/qbzr/commit.py", line 220, in __init__
    for entry in tree.unknowns():
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/workingtree.py", line 1510, in unknowns
    [subp for subp in self.extras() if not self.is_ignored(subp)])
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/workingtree.py", line 1640, in extras
    subf_norm, can_access = osutils.normalized_filename(subf)
  File "/home/lukas/projects/bzr/bzr.dev/bzrlib/osutils.py", line 1030, in _inaccessible_normalized_filename
    normalized = unicodedata.normalize('NFKC', unicode(path))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe1 in position 0: ordinal not in range(128)

but since QBzr explicitly asks for unknown files, I'm not sure what would be the right way to work around this problem.

[Robert Collins (lifeless)]

 status confirmed
 importance low

(low because there is an obvious workaround ::)

-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.

[Daniel Hahler (blueyed)]

The same happens for "bzr status", too.

It would be nice, if that would not fail, because in this case I have a bunch of files, which I do not want to add, but would like bzr to ignore them - so that I can use "bzr status".

$ cd /tmp
$ mkdir foobar
$ bzr init
$ touch $(php -r 'echo "w\344schekammer_01.jpg";')
$ ls
w?schekammer_01.jpg
$ bzr status
bzr: ERROR: exceptions.UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-3: invalid data

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 800, in ignore_pipe
    result = func(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 189, in run
    to_file=self.outf, short=short, versioned=versioned)
  File "/usr/lib/python2.5/site-packages/bzrlib/status.py", line 114, in show_tree_status
    want_unversioned=want_unversioned)
  File "/usr/lib/python2.5/site-packages/bzrlib/tree.py", line 93, in changes_from
    want_unversioned=want_unversioned,
  File "/usr/lib/python2.5/site-packages/bzrlib/decorators.py", line 127, in read_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/tree.py", line 727, in compare
    want_unversioned=want_unversioned)
  File "/usr/lib/python2.5/site-packages/bzrlib/delta.py", line 217, in _compare_trees
    want_unversioned=want_unversioned):
  File "/usr/lib/python2.5/site-packages/bzrlib/workingtree_4.py", line 2418, in iter_changes
    (None, utf8_decode(current_path_info[0])[0]),
  File "/usr/lib/python2.5/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1-3: invalid data

bzr 1.3.1 on python 2.5.2.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'st']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'de_DE.UTF-8'
plugins:
  builddeb /usr/lib/python2.5/site-packages/bzrlib/plugins/builddeb [0.92.0dev0]
  bzrtools /usr/lib/python2.5/site-packages/bzrlib/plugins/bzrtools [1.3.0]
  gtk /usr/lib/python2.5/site-packages/bzrlib/plugins/gtk [0.94.0]
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
  qbzr /usr/lib/python2.5/site-packages/bzrlib/plugins/qbzr [0.9.0]
  rebase /usr/lib/python2.5/site-packages/bzrlib/plugins/rebase [0.3.0]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[Pierre (pierre-beaumadier)]

I have the same issue