crash with winmerge in cp932 japanese character

[Heart-Rhythm]

bzr: ERROR: exceptions.UnicodeDecodeError: 'cp932' codec can't decode bytes in position 26-27: illegal multibyte sequence

Traceback (most recent call last):
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 605, in readStdout
UnicodeDecodeError: 'cp932' codec can't decode bytes in position 26-27: illegal multibyte sequence

bzr 2.1.0 on python 2.5.4 (win32)
arguments: ['C:\\Program Files\\Bazaar\\bzr.exe', '--command=diff', '--file=C:\\TEMP\\bzrtest-bugfix\\\x83_\x83~\x81[\x95\\\x83h\x83L\x83\x85\x83\x81\x83\x93\x83g.txt']
encoding: 'cp932', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Program Files\Bazaar\plugins\explorer [1.0.0rc1]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.0]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.1.0]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.18.1]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.2]
  upload C:\Program Files\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Bazaar\plugins\xmloutput [0.8.6]

[Alexander Belchenko (bialix)]

Naoki, I need your help on understanding what's going on.

[methane (songofacandy)]

Heart-Rhythm　さん、バグ報告ありがとうございます。
問題解決のために、問題発生時の詳しい状況を教えていただけますか？

1. WinMerge は Unicode版 (WinMergeU) でしょうか？ ASCII 版でしょうか？

2. どういった操作をしたときにこのエラーが発生したのでしょうか？

3. スタートメニューから "Open bzr log file" を選ぶと、詳細なログが表示されるので、
   該当するエラーの周辺を添付していただけますか？

[methane (songofacandy)]

すみません、試してみたら簡単に再現できました。

2010/2/18 INADA Naoki <email address hidden>:
> Heart-Rhythm　さん、バグ報告ありがとうございます。
> 問題解決のために、問題発生時の詳しい状況を教えていただけますか？
>
> 1. WinMerge は Unicode版 (WinMergeU) でしょうか？ ASCII 版でしょうか？
>
> 2. どういった操作をしたときにこのエラーが発生したのでしょうか？
>
> 3. スタートメニューから "Open bzr log file" を選ぶと、詳細なログが表示されるので、
> 該当するエラーの周辺を添付していただけますか？
>

--
INADA Naoki <email address hidden>

[methane (songofacandy)]

I success to reproduce this bug:

C:\test2>bzr init
Created a standalone tree (format: 2a)

C:\test2>echo abc > あいう

C:\test2>bzr add あいう
adding "あいう"

C:\test2>bzr commit -m "abc"
Committing to: C:/test2/
added あいう
Committed revision 1.

C:\test2>echo def >> あいう

C:\test2>bzr diff
=== modified file '縺ゅ＞縺・
--- 縺ゅ＞縺・2010-02-18 12:12:23 +0000
+++ 縺ゅ＞縺・2010-02-18 12:12:40 +0000
@@ -1,1 +1,2 @@
 abc
+def

C:\test2>bzr diff --using=C:\\usr\\WinMerge\\WinMergeU.exe
=== modified file '縺ゅ＞縺・
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode characters in position 84-86: ordinal not in r
(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\commands.pyo", line 1070, in ignore_pipe
  File "bzrlib\builtins.pyo", line 1955, in run
  File "bzrlib\diff.pyo", line 444, in show_diff_trees
  File "bzrlib\diff.pyo", line 889, in show_diff
  File "bzrlib\diff.pyo", line 949, in _show_diff
  File "bzrlib\diff.pyo", line 975, in _diff
  File "bzrlib\diff.pyo", line 506, in _diff_many
  File "bzrlib\diff.pyo", line 782, in diff
  File "bzrlib\diff.pyo", line 708, in _execute
  File "subprocess.pyo", line 594, in __init__
  File "subprocess.pyo", line 822, in _execute_child
UnicodeEncodeError: 'ascii' codec can't encode characters in position 84-86: ordinal not in range(128)

bzr 2.1.0 on python 2.5.4 (Windows-XP-5.1.2600-SP3)
arguments: ['bzr', 'diff', '--using=C:\\\\usr\\\\WinMerge\\\\WinMergeU.exe']
encoding: 'cp932', fsenc: 'mbcs', lang: 'ja'
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Documents and Settings\inada-n\Application Data\bazaar\2.0\plugins\explorer [1.0.0.rc.0]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.0]
  loom C:\Documents and Settings\inada-n\Application Data\bazaar\2.0\plugins\loom [1.4.0dev]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.1.0]
  qbzr C:\Documents and Settings\inada-n\Application Data\bazaar\2.0\plugins\qbzr [0.19.0dev]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  scmproj C:\Documents and Settings\inada-n\Application Data\bazaar\2.0\plugins\scmproj [0.6.0dev]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.2]
  symlink C:\Documents and Settings\inada-n\Application Data\bazaar\2.0\plugins\symlink [0.92.0]
  upload C:\Program Files\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Bazaar\plugins\xmloutput [0.8.6]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

Traceback is:
0.093 bazaar version: 2.1.0
0....

[methane (songofacandy)]

I can reproduce this bug without qbzr.

C:\test2>bzr --no-plugins diff --using=C:\\usr\\WinMerge\\WinMergeU.exe
=== modified file '縺ゅ＞縺・
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode characters in position 84-86: ordinal not in range
(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\commands.pyo", line 1070, in ignore_pipe
  File "bzrlib\builtins.pyo", line 1955, in run
  File "bzrlib\diff.pyo", line 444, in show_diff_trees
  File "bzrlib\diff.pyo", line 889, in show_diff
  File "bzrlib\diff.pyo", line 949, in _show_diff
  File "bzrlib\diff.pyo", line 975, in _diff
  File "bzrlib\diff.pyo", line 506, in _diff_many
  File "bzrlib\diff.pyo", line 782, in diff
  File "bzrlib\diff.pyo", line 708, in _execute
  File "subprocess.pyo", line 594, in __init__
  File "subprocess.pyo", line 822, in _execute_child
UnicodeEncodeError: 'ascii' codec can't encode characters in position 84-86: ordinal not in range(128)

bzr 2.1.0 on python 2.5.4 (Windows-XP-5.1.2600-SP3)
arguments: ['bzr', '--no-plugins', 'diff', '--using=C:\\\\usr\\\\WinMerge\\\\WinMergeU.exe']
encoding: 'cp932', fsenc: 'mbcs', lang: 'ja'
plugins:

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it by filing a bug report at
        https://bugs.launchpad.net/bzr/+filebug
    including this traceback and a description of the problem.

[methane (songofacandy)]

OK, There are three problems.

1. bzr diff shows filename in "utf-8" encoding.
    This is not bad if stdout redirected to file. But if stdout is console, bzr should encode with stdout.encoding.

2. bzr diff --using=prog fails when arguments contains non ascii char.

3. qbzr crashes when qsubprocess writes binary to stdout.
    I think qbzr should use decode with errors="replace" or errors="backslashreplace".

[Alexander Belchenko (bialix)]

Heart-Rhythm пишет:
> Public bug reported:
>
> bzr: ERROR: exceptions.UnicodeDecodeError: 'cp932' codec can't decode
> bytes in position 26-27: illegal multibyte sequence
>
> Traceback (most recent call last):
> File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 605, in readStdout
> UnicodeDecodeError: 'cp932' codec can't decode bytes in position 26-27: illegal multibyte sequence
>
> bzr 2.1.0 on python 2.5.4 (win32)
> arguments: ['C:\\Program Files\\Bazaar\\bzr.exe', '--command=diff', '--file=C:\\TEMP\\bzrtest-bugfix\\\x83_\x83~\x81[\x95\\\x83h\x83L\x83\x85\x83\x81\x83\x93\x83g.txt']
> encoding: 'cp932', fsenc: 'mbcs', lang: None

Please, show me relevant part of .bzr.log. There should be more info
about this problem.

[Alexander Belchenko (bialix)]

Thanks, Naoki.

[methane (songofacandy)]

This is a patch to fix problem 2.

BTW, Should we split this bug to 3 bugs?

[methane (songofacandy)]

This patch fixes problem 3.

[methane (songofacandy)]

I can't fix problem 1 because I don't know about ui well.
How am I able to determine outf is redirected?

[methane (songofacandy)]

I've committed fixing problem 2 to lp:qbzr and lp:qbzr/0.18.

[Alexander Belchenko (bialix)]

Naoki, fixing problem 1 should be done in bzr itself. There is bug report about utf-8.

But unfortunately we have no way to detect if we redirect to file or simply run `bzr command|less`.

Thanks for your patches.

[methane (songofacandy)]

Should I make a branch from bzr-2.0?

[methane (songofacandy)]

I've reimplemented fix and propose merge request.

[Martin Packman (gz)]

The branch addressing this problem has landed! Some additional manual testing may be useful here to ensure there aren't any regressions.