unicode handling problem

[Dada Priyatosh]

It seems that there is a problem with files with utf-8 content.

D:\web\htdocs\d6\sites\all\modules\travmap>bzr export --keywords=publish d:/111
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 359: ordinal not in range(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 842, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1037, in run_bzr
  File "bzrlib\commands.pyo", line 654, in run_argv_aliases
  File "C:/Users/1/AppData/Roaming/bazaar/2.0/plugins\keywords\__init__.py", line 451, in run
  File "bzrlib\builtins.pyo", line 2744, in run
  File "bzrlib\export\__init__.pyo", line 102, in export
  File "bzrlib\export\__init__.pyo", line 61, in _loader
  File "bzrlib\export\dir_exporter.pyo", line 62, in dir_exporter
  File "bzrlib\filters\__init__.pyo", line 170, in filtered_output_bytes
  File "C:/Users/1/AppData/Roaming/bazaar/2.0/plugins\keywords\__init__.py", line 366, in _normal_kw_expander
  File "C:/Users/1/AppData/Roaming/bazaar/2.0/plugins\keywords\__init__.py", line 361, in _kw_expander
  File "C:/Users/1/AppData/Roaming/bazaar/2.0/plugins\keywords\__init__.py", line 329, in expand_keywords
UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 359: ordinal not in range(128)

bzr 2.0.0 on python 2.5.4 (Windows-32bit-SP1)
arguments: ['bzr', 'export', '--keywords=publish', 'd:/111']
encoding: 'cp1251', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.0.0]
  explorer C:\Program Files\Bazaar\plugins\explorer [0.8.2]
  keywords C:\Users\1\AppData\Roaming\bazaar\2.0\plugins\keywords [unknown]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.0.0]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.0.0]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.14.2]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.3]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.0]
  upload C:\Program Files\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Bazaar\plugins\xmloutput [0.8.5]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[Ryan Dortmans (ryan-dortmans)]

I also have the same issue. It seems to be able to handle some files but not others.

To reproduce, I created a file with the following contents:

// $Filename$

®

I added this to a test repository and committed. Then ran the following:

bzr export --filters /bzr/temp/export ./

OR

bzr cat --filters testfile.cpp

Both end up with the following error:

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 2: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 927, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 1127, in run_bzr
    ret = run(*run_argv)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 692, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/usr/lib/python2.6/site-packages/bzrlib/commands.py", line 714, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/plugins/keywords/__init__.py", line 223, in run
    return super(cmd_export, self).run(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/bzrlib/builtins.py", line 2893, in run
    per_file_timestamps=per_file_timestamps)
  File "/usr/lib/python2.6/site-packages/bzrlib/export/__init__.py", line 109, in export
    per_file_timestamps=per_file_timestamps)
  File "/usr/lib/python2.6/site-packages/bzrlib/export/__init__.py", line 63, in _loader
    per_file_timestamps=per_file_timestamps)
  File "/usr/lib/python2.6/site-packages/bzrlib/export/dir_exporter.py", line 84, in dir_exporter
    chunks = filtered_output_bytes(chunks, filters, context)
  File "/usr/lib/python2.6/site-packages/bzrlib/filters/__init__.py", line 170, in filtered_output_bytes
    chunks = filter.writer(chunks, context)
  File "/usr/lib/python2.6/site-packages/bzrlib/plugins/keywords/keywords.py", line 259, in _normal_kw_expander
    return _kw_expander(chunks, context)
  File "/usr/lib/python2.6/site-packages/bzrlib/plugins/keywords/keywords.py", line 254, in _kw_expander
    encoder=encoder)]
  File "/usr/lib/python2.6/site-packages/bzrlib/plugins/keywords/keywords.py", line 221, in expand_keywords
    return result + rest
UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 2: ordinal not in range(128)

[Ryan Dortmans (ryan-dortmans)]

I should also mention that this bug prevents a repository containing a file with this character from being checked out when the plugin is enabled.