unicode command line options cause unicodeencodeerror traceback

[Martin Pool]

Bazaar gives a traceback if a unicode character is given in a command option:

mbp@grace% ./bzr st -ة
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0629' in position 17: ordinal not in range(128)

Traceback (most recent call last):
  File "/home/mbp/bzr/trivial/bzrlib/commands.py", line 802, in run_bzr_catch_errors
    return run_bzr(argv)
  File "/home/mbp/bzr/trivial/bzrlib/commands.py", line 758, in run_bzr
    ret = run(*run_argv)
  File "/home/mbp/bzr/trivial/bzrlib/commands.py", line 465, in run_argv_aliases
    args, opts = parse_args(self, argv, alias_argv)
  File "/home/mbp/bzr/trivial/bzrlib/commands.py", line 544, in parse_args
    options, args = parser.parse_args(args)
  File "optparse.py", line 1380, in parse_args
    self.error(str(err))
UnicodeEncodeError: 'ascii' codec can't encode character u'\u0629' in position 17: ordinal not in range(128)

This is similar to bug 57123, but probably recurring because we have switched to using optparse since that commit was merged. Unlike that bug, this only happens with options, not command names.

[Martin Pool (mbp)]

Test user <email address hidden> <email address hidden>

[Martin Pool (mbp)]

Originally reported by "Grinchishin V." <email address hidden>

[John A Meinel (jameinel)]

Interestingly, when you view this page without being logged in, you get <email address removed> instead of <email address hidden>.

Anyway, I just wanted to mention that I believe this is actually a bug in OptParse, in that it cannot handle Unicode options.

I'm not sure if there is already a bug on it, but I ran into that when trying to test some unicode command line functionality. (Like having i18n commands and options.)

So it is a bug in an upstream library, I'm not sure how we want to work around it.

[Martin Pool (mbp)]

> Interestingly, when you view this page without being logged in, you get <email address removed> instead of <email address hidden>.

That's the intended behavior; Launchpad is meant to hide email addresses from unauthenticated users.

For this particular bug I think it'd be reasonable to just catch unicodeerror from the parse statement and display a message.

[Martin Pool (mbp)]

See also the traceback in bug 463052

[Alexey Gaynulin (ag-bernoulli)]

I don't understand how cn i solve this problem:
bzr: ERROR: exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\u0438' in position 0: ordinal not in range(128)

[Alexey Gaynulin (ag-bernoulli)]

This is really problem - i can't add something, merge and push files and changes on the server..

[Parth Malwankar (parthm)]

Hi Alexey,

What is the exact command your are trying to run? Also could you please paste the trace thats printed on the screen on failure.

[Alexey Gaynulin (ag-bernoulli)]

As a variant: for example i do the following:

Run command: bzr init-repository --no-trees sftp://bazaar.09248.ru/bzr
Connected (version 2.0, client OpenSSH_4.3)

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 4: ordinal not in range(128)

Traceback (most recent call last):
  File "bzrlib\commands.pyo", line 853, in exception_to_return_code
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\commands.py", line 788, in run
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 786, in run_subprocess_command
  File "bzrlib\commands.pyo", line 1055, in run_bzr
  File "bzrlib\commands.pyo", line 661, in run_argv_aliases
  File "bzrlib\commands.pyo", line 665, in run_direct
  File "bzrlib\cleanup.pyo", line 122, in run_simple
  File "bzrlib\cleanup.pyo", line 156, in _do_with_cleanups
  File "bzrlib\builtins.pyo", line 1817, in run
  File "bzrlib\transport\__init__.pyo", line 368, in ensure_base
  File "bzrlib\transport\sftp.pyo", line 662, in mkdir
  File "bzrlib\transport\sftp.pyo", line 638, in _mkdir
  File "bzrlib\transport\sftp.pyo", line 397, in _get_sftp
  File "bzrlib\transport\sftp.pyo", line 389, in _create_connection
  File "bzrlib\transport\ssh.pyo", line 330, in connect_sftp
  File "bzrlib\transport\ssh.pyo", line 326, in _connect
  File "bzrlib\transport\ssh.pyo", line 540, in _paramiko_auth
  File "bzrlib\config.pyo", line 1288, in get_password
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 735, in get_password
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 726, in _get_answer_from_main
  File "C:/Program Files/Bazaar/plugins\qbzr\lib\subprocess.py", line 847, in bencode_prompt
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 4: ordinal not in range(128)

bzr 2.1.1 on python 2.5.4 (Windows-Vista-6.1.7600)
arguments: ['C:\\Program Files\\Bazaar\\bzr.exe', 'qsubprocess', '--bencode', 'l15:init-repository10:--no-trees26:sftp://bazaar.09248.ru/bzre']
encoding: 'cp1251', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files\Bazaar\plugins\bzrtools [2.1.0]
  explorer C:\Program Files\Bazaar\plugins\explorer [1.0.1]
  launchpad C:\Program Files\Bazaar\plugins\launchpad [2.1.1]
  netrc_credential_store C:\Program Files\Bazaar\plugins\netrc_credential_store [2.1.1]
  news_merge C:\Program Files\Bazaar\plugins\news_merge [2.1.1]
  qbzr C:\Program Files\Bazaar\plugins\qbzr [0.18.4]
  rebase C:\Program Files\Bazaar\plugins\rebase [0.5.5]
  svn C:\Program Files\Bazaar\plugins\svn [1.0.2]
  upload C:\Program Files\Bazaar\plugins\upload [1.0.0dev]
  xmloutput C:\Program Files\Bazaar\plugins\xmloutput [0.8.7.dev]

*** Bazaar has encountered an internal error. This probably indicates a
    bug in Bazaar. You can help us fix it ...

[Parth Malwankar (parthm)]

Hi Alexey,

From the trace it looks that the command line options are being processed correctly. This looks like a separate bug, UnicodeEncodeError is kind of a catch all error. Could you file a separate report with the above details please. This will allow the issue to be tracked better.

The crash seems to be happening in qbzr plugin. Does this work from the command prompt?

[Alexey Gaynulin (ag-bernoulli)]

Parth,
i've been trying to use Bazaar from yesterday, so ...
"Could you file a separate report with the above details please. "
- you mean to copy the error-trace in file and send...?!

"Does this work from the command prompt?"
- as i tried in "Start BZR in Cmd shell" - thr similar error with same text is caused.
http://clip2net.com/page/m0/6273896 (<- this is screen shot)
This error prevents me to work with server. I can do actions on the local machine without any problems, but when i push something to the server or work centralized (constant connetction) with server - every time occur thi problem ((

[Alexander Belchenko (bialix)]

Alexey, please do following:

1) File a *new* bug report against QBzr project for your problem

2) run the following command:

bzr qsubprocess init-repository --no-trees sftp://bazaar.09248.ru/bzr >stdout 2>stderr

and attach both stdout and stderr files to the *new* bug report

[Alexander Belchenko (bialix)]

2Alexey: sorry, typo here, about running command. Please run the followiung command:

bzr qsubprocess "init-repository --no-trees sftp://bazaar.09248.ru/bzr" >stdout 2>stderr

[Alexander Belchenko (bialix)]

2Alexey: ok, based on the screenshot you have provided, this is actually bug in bzr, not in qbzr.

So, again:

1) File *new* bug report. Provide description of what you're doing and error message.

2) To copy the text from the console you don't need a screenshot. Leftt-click on console shell window icon (left-top), select Edit -> Mark (Изменить -> Выделить). Draw a rectangle with mouse over the text you want to copy and press Enter. Then paste this text to editor or bug report with Ctrl+C.

3) Add the relevant part of .bzr.log to the bug report. You can find the location of .bzr.log file by running command `bzr version`.

[Alexey Gaynulin (ag-bernoulli)]

Alexander, where should i send my attached file to?

[Alexey Gaynulin (ag-bernoulli)]

the logs' and errors' reports are in the .txt file - where can i send it?
Or post it here in textarea??

[Parth Malwankar (parthm)]

Hi Alexey,

When you file a new report (at https://bugs.launchpad.net/bzr/+filebug ), at the end of every bug report there is a "Add attachment or patch" link. This can be used to attach additional files to the bug report.

[Alexey Gaynulin (ag-bernoulli)]

Sorry, have't noticed before.
 Even made new post.

The logs and errors reports are attached