bzr add on Unicode filename fails

[John Whitley]

Repro steps:
1) bzr init test
2) cd test
3) touch file.txt
4) Use Windows Explorer / Finder to rename file.txt to REQUÊTE.TXT
5) bzr add .

Cygwin results:
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 5: ordinal not in range(128)
  at /tmp/python.572/usr/lib/python2.4/posixpath.py line 65
  in join

Mac OS X results:
added bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0302' in position 6: ordinal not in range(128)
  at /usr/local/lib/bzr-0.8/bzrlib/add.py line 61
  in add_action_print

Note that the way the unicode name was generated on each system was slightly different -- the Cygwin version was from a file we have in CVS, while I manually retyped the version on Mac OS X using option-i E to get 'Ê' on a US keyboard.

[John Whitley (whitley)]

Missed some platform/version notes:
Platforms: Cygwin and Mac OS X 10.4.6 (PowerPC)
bzr version: 0.8 release

[John Whitley (whitley)]

I'm still able to repro this bug on Cygwin now that JAM's encodings work has landed. Mac OS X appears to work fine. I used bzr.dev revno 1816 for this trial. The exception thrown is much the same:

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 5: ordinal not in range(128)

Traceback (most recent call last):
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/commands.py", line 692, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/commands.py", line 655, in run_bzr
    ret = run(*run_argv)
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/commands.py", line 281, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/builtins.py", line 276, in run
    action=action, save=not dry_run)
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/add.py", line 104, in smart_add
    return smart_add_tree(tree, file_list, recurse, action=action)
  File "/home/jwhitley/src/bzr/no-knit-cache/bzrlib/add.py", line 252, in smart_add_tree
    subp = bzrlib.osutils.pathjoin(directory.raw_path, subf)
  File "/tmp/python.340/usr/lib/python2.4/posixpath.py", line 65, in join
    path += '/' + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0xca in position 5: ordinal not in range(128)

bzr 0.9.0dev0 on python 2.4.3.final.0 (cygwin)
arguments: ['/home/jwhitley/local/bin/bzr', 'add']

[John A Meinel (jameinel)]

I'm pretty sure this is fixed in the latest bzr.dev. My 'mac' updates may break something, because of mac normalization issues.
(Files created and added on Linux will show up under a different path on Mac, and now I disallow the alternate normalization. I fixed our reader for Mac, I'm not sure what steps are necessary to fix a branch that might be broken. If this is an issue, ping me and we'll get a fix)
For now, considering this bug closed.

[Mat Kellogg (mathkellogg)]

mat@comp:$ bzr add
added Utils\db\10.10.1.186Ping.shlf
bzr: ERROR: bzrlib.errors.InvalidEntryName: Invalid entry name: Utils\db\10.10.1.186Ping.shlf

Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 834, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 790, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.5/site-packages/bzrlib/commands.py", line 492, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.5/site-packages/bzrlib/builtins.py", line 384, in run
    no_recurse, action=action, save=not dry_run)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 51, in tree_write_locked
    return unbound(self, *args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 384, in smart_add
    _add_one(self, inv, parent_ie, directory, kind, action)
  File "/usr/lib/python2.5/site-packages/bzrlib/mutabletree.py", line 534, in _add_one
    file_id=file_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 1083, in make_entry
    return make_entry(kind, name, parent_id, file_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 1417, in make_entry
    return factory(file_id, name, parent_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 642, in __init__
    super(InventoryFile, self).__init__(file_id, name, parent_id)
  File "/usr/lib/python2.5/site-packages/bzrlib/inventory.py", line 308, in __init__
    raise errors.InvalidEntryName(name=name)
InvalidEntryName: Invalid entry name: Utils\db\10.10.1.186Ping.shlf

bzr 1.3.1 on python 2.5.2.final.0 (linux2)
arguments: ['/usr/bin/bzr', 'add']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'en_US.UTF-8'
plugins:
  launchpad /usr/lib/python2.5/site-packages/bzrlib/plugins/launchpad [unknown]
*** Bazaar has encountered an internal error.
    Please report a bug at https://bugs.launchpad.net/bzr/+filebug
    including this traceback, and a description of what you
    were doing when the error occurred.

[fms (fms-silerfamily)]

% bzr add
[snip okay output]
adding "Benny Goodman"
adding "Béla Fleck"
bzr: ERROR: exceptions.KeyError: u'Be\u0301la Fleck'

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", lin
e 912, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", lin
e 1112, in run_bzr
    ret = run(*run_argv)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", lin
e 690, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/commands.py", lin
e 705, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/cleanup.py", line
 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/cleanup.py", line
 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/builtins.py", lin
e 690, in run
    no_recurse, action=action, save=not dry_run)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/mutabletree.py",
line 50, in tree_write_locked
    return unbound(self, *args, **kwargs)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/mutabletree.py",
line 538, in smart_add
    this_ie = parent_ie.children[directory.base_path]
KeyError: u'Be\u0301la Fleck'

bzr 2.2.1 on python 2.6.6 (Darwin-10.4.0-i386-64bit)
arguments: ['/opt/local/bin/bzr', 'add']
encoding: 'UTF-8', fsenc: 'utf-8', lang: 'en_US.UTF-8'
plugins:
  bash_completion /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/pl
ugins/bash_completion [2.2.1]
  launchpad /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/pl
ugins/launchpad [2.2.1]
  netrc_credential_store /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/
plugins/netrc_credential_store [2.2.1]
  news_merge /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib/pl
ugins/news_merge [2.2.1]

[fms (fms-silerfamily)]

Sorry, forgot to add: I'm running MacOS 10.6 with bzr installed from MacPorts, latest versions.

% uname -a
Darwin Birch.local 10.4.0 Darwin Kernel Version 10.4.0: Fri Apr 23 18:28:53 PDT 2010; root:xnu-1504.7.4~1/RELEASE_I386 i
386 i386 MacBookPro5,5 Darwin

% bzr --version
Bazaar (bzr) 2.2.1
  Python interpreter: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Py
thon 2.6.6
  Python standard library: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6
  Platform: Darwin-10.4.0-i386-64bit
  bzrlib: /opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/bzrlib
  Bazaar configuration: /Users/fms/.bazaar
  Bazaar log file: /Users/fms/.bzr.log

Copyright 2005-2010 Canonical Ltd.
http://bazaar.canonical.com/

bzr comes with ABSOLUTELY NO WARRANTY. bzr is free software, and
you may use, modify and redistribute it under the terms of the GNU
General Public License version 2 or later.

Bazaar is part of the GNU Project to produce a free operating system.