`unshelve --preview` fails with unicode error

[Alexander Belchenko]

Preview of unshelving file with non-ascii characters in it failed with traceback:

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc8 in position 3: ordinal not in range(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 5820, in run
  File "bzrlib\shelf_ui.pyo", line 481, in run
  File "bzrlib\shelf_ui.pyo", line 500, in write_diff
  File "bzrlib\diff.pyo", line 444, in show_diff_trees
  File "bzrlib\diff.pyo", line 889, in show_diff
  File "bzrlib\diff.pyo", line 949, in _show_diff
  File "bzrlib\diff.pyo", line 975, in _diff
  File "bzrlib\diff.pyo", line 506, in _diff_many
  File "bzrlib\diff.pyo", line 644, in diff
  File "bzrlib\diff.pyo", line 667, in diff_text
  File "bzrlib\diff.pyo", line 113, in internal_diff
  File "bzrlib\ui\text.pyo", line 488, in write
  File "codecs.pyo", line 303, in write
  File "encodings\cp866.pyo", line 12, in encode
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc8 in position 3: ordinal not in range(128)

bzr 2.1.0rc1 on python 2.5.4 (Windows-XP-5.1.2600-SP3)
arguments: ['C:\\Program Files\\Bazaar\\bzr.EXE', 'unshelve', '--preview']
encoding: 'cp1251', fsenc: 'mbcs', lang: None
plugins:
  acad C:\work\Bazaar\plugins\acad [0.8.0]
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0b1]
  colo C:\work\Bazaar\plugins\colo [0.0.2dev]
  explorer C:\work\Bazaar\plugins\explorer [1.0.0.rc.0]
  fastimport C:\work\Bazaar\plugins\fastimport [0.9.0dev]
  format1 C:\work\Bazaar\plugins\format1 [unknown]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.0rc1]
  qbzr C:\work\Bazaar\plugins\qbzr [0.19.0dev]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  scmproj C:\work\Bazaar\plugins\scmproj [0.5.2dev]
  x_bit C:\work\Bazaar\plugins\x_bit [1.0.0]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

I'm suspect the problem here is the same as with log -p: unshelve using wrong encoding mode for self.outf. The same approach should be applied here: to print the diff raw stdout stream should be used.

[Martin Pool (mbp)]

I'd say that in our new approach, it should be asking for a binary-mode make_output_stream().

[Alexander Belchenko (bialix)]

Martin Pool пишет:
> I'd say that in our new approach, it should be asking for a binary-mode
> make_output_stream().

No, I disagree. If there is unicode filenames they should be printed in terminal encoding. But the
diff itself should be emitted in binary mode.

Output stream should provides both unicode aware stream and binary safe stream.

[Ralf Doering (rdoering)]

The same happened here, crashlog from bzr unshelve --preview is attached.

[Alexander Belchenko (bialix)]

Looking inside shelf file it seems bzr stores filenames as utf-8 bytestreams. Thus it should decode them back to unicode on unshelve. Maybe this is the reason of the bug.

[Alexander Belchenko (bialix)]

Another traceback:

Using changes with id "1".
 M тест.txt
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xf2 in position 19: ordinal not in range(128)

Traceback (most recent call last):
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\commands.py", line 853, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\commands.py", line 1055, in run_bzr
    ret = run(*run_argv)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\commands.py", line 661, in run_argv_aliases
    return self.run_direct(**all_cmd_args)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\commands.py", line 665, in run_direct
    return self._operation.run_simple(*args, **kwargs)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\cleanup.py", line 122, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\cleanup.py", line 156, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\builtins.py", line 5829, in run
    unshelver.run()
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\shelf_ui.py", line 482, in run
    self.write_diff(merger)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\shelf_ui.py", line 505, in write_diff
    path_encoding=path_encoding)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\diff.py", line 444, in show_diff_trees
    return differ.show_diff(specific_files, extra_trees)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\diff.py", line 892, in show_diff
    return self._show_diff(specific_files, extra_trees)
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\diff.py", line 950, in _show_diff
    newpath_encoded, prop_str))
  File "C:\work\Bazaar\bzr-2a\2.1\bzrlib\ui\text.py", line 504, in write
    self.wrapped_stream.write(to_write)
  File "C:\Python25\lib\codecs.py", line 303, in write
    data, consumed = self.encode(object, self.errors)
  File "C:\Python25\lib\encodings\cp1251.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_table)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf2 in position 19: ordinal not in range(128)

[Andrej A Antonov (polymorphm)]

not sure what the true-root error is corrected, but this fix behaves seems correctly

https://code.launchpad.net/~polymorphm/bzr/BUG_FIX_518916

(но ещё не тестировал на Microsoft Windows)

[Andrej A Antonov (polymorphm)]

(Not yet tested on Microsoft Windows)

[Andrej A Antonov (polymorphm)]

sorry for flud&spam :-( :-( -- code.launchpad.net/~polymorphm/bzr/BUG_FIX_518916 -- is not an option

regressions on operations "$ bzr cat ..." . in MSWindows and GNU/Linux

[Andrej A Antonov (polymorphm)]

in file 'bzrlib/shelf_ui.py' -- output stream created with invalid parameters