UnicodeError while converting BzrError to unicode

[jean-michel]

I installed Bazaar on an Ubuntu 10.10 server.
I created a test repository in /srv/bzr/bazaar.test.
In that directory I initialised a branch called "trunk". In that branch I created a file called "first.file.txt" that contain some simple text. I added the file and commited the change.

Then when trying to open that URL from Bazaar Explorer on my desktop (French Windows 7 sp1), using the open URL with bzr://ubuntu-srv-1.kzo-corp.com/svr/bzr/test.bazaar/trunk, I get the following erreor:

bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position 89: ordinal not in range(128)

Traceback (most recent call last):
  File "C:/Program Files (x86)/Bazaar/plugins\explorer\lib\explorer.py", line 1448, in callable_with_cleanup
  File "C:/Program Files (x86)/Bazaar/plugins\explorer\lib\explorer.py", line 1614, in _open_location_task
  File "C:/Program Files (x86)/Bazaar/plugins\explorer\lib\explorer.py", line 1646, in _show_unable_to_open_location
  File "bzrlib\errors.pyo", line 120, in __unicode__
UnicodeDecodeError: 'ascii' codec can't decode byte 0x92 in position 89: ordinal not in range(128)

bzr 2.3.0 on python 2.6.6 (win32)
arguments: ['C:\\Program Files (x86)\\Bazaar\\bzrw.exe', 'explorer']
encoding: 'cp1252', fsenc: 'mbcs', lang: None
plugins:
  bzrtools C:\Program Files (x86)\Bazaar\plugins\bzrtools [2.3.1]
  colo C:\Program Files (x86)\Bazaar\plugins\colo [0.2.1]
  explorer C:\Program Files (x86)\Bazaar\plugins\explorer [1.1.2]
  fastimport C:\Program Files (x86)\Bazaar\plugins\fastimport [0.10.0dev]
  launchpad C:\Program Files (x86)\Bazaar\plugins\launchpad [2.3.0]
  loom C:\Program Files (x86)\Bazaar\plugins\loom [2.2.1dev]
  netrc_credential_store C:\Program Files (x86)\Bazaar\plugins\netrc_credential_store [2.3.0]
  news_merge C:\Program Files (x86)\Bazaar\plugins\news_merge [2.3.0]
  pipeline C:\Program Files (x86)\Bazaar\plugins\pipeline [1.1.0]
  qbzr C:\Program Files (x86)\Bazaar\plugins\qbzr [0.20.0]
  rewrite C:\Program Files (x86)\Bazaar\plugins\rewrite [0.6.2dev]
  svn C:\Program Files (x86)\Bazaar\plugins\svn [1.0.5dev]
  upload C:\Program Files (x86)\Bazaar\plugins\upload [1.0.0]
  xmloutput C:\Program Files (x86)\Bazaar\plugins\xmloutput [0.8.7.dev]

[jean-michel (jeanmichel-cazaux)]

And when trying the same thing from an Ubuntu desktop, I get the same error.

Exact error:
bzr: ERROR: exceptions.UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 86: ordinal not in range(128)

Traceback (most recent call last):
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/explorer/lib/explorer.py", line 1450, in callable_with_cleanup
    task_callable()
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/explorer/lib/explorer.py", line 1616, in _open_location_task
    auto_virtual_repo_open)
  File "/usr/lib/python2.6/dist-packages/bzrlib/plugins/explorer/lib/explorer.py", line 1648, in _show_unable_to_open_location
    why = gettext(unicode(ex))
  File "/usr/lib/python2.6/dist-packages/bzrlib/errors.py", line 120, in __unicode__
    u = unicode(u)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 86: ordinal not in range(128)

bzr 2.2.1 on python 2.6.6 (linux2)
arguments: ['/usr/bin/bzr', 'explorer']
encoding: 'UTF-8', fsenc: 'UTF-8', lang: 'fr_FR.utf8'
plugins:
  bash_completion /usr/lib/python2.6/dist-packages/bzrlib/plugins/bash_completion [2.2.1]
  bzrtools /usr/lib/python2.6/dist-packages/bzrlib/plugins/bzrtools [2.2.0]
  explorer /usr/lib/python2.6/dist-packages/bzrlib/plugins/explorer [1.1.0]
  launchpad /usr/lib/python2.6/dist-packages/bzrlib/plugins/launchpad [2.2.1]
  netrc_credential_store /usr/lib/python2.6/dist-packages/bzrlib/plugins/netrc_credential_store [2.2.1]
  news_merge /usr/lib/python2.6/dist-packages/bzrlib/plugins/news_merge [2.2.1]
  qbzr /usr/lib/python2.6/dist-packages/bzrlib/plugins/qbzr [0.1

[Alexander Belchenko (bialix)]

I don't see any non-ascii characters in your URL: bzr://ubuntu-srv-1.kzo-corp.com/svr/bzr/test.bazaar/trunk

Do you have any ideas what is could be? Maybe your username?

Also, you're trying to open bzr:// URL. Do you have the `bzr serve` running on your Ubuntu machine? If so, can you show the relevant part of .bzr.log from the client side and from the server side?

[Alexander Belchenko (bialix)]

Ah sorry, I see now. We're trying to convert localized error coming from OS to unicode and it fails.

[Alexander Belchenko (bialix)]

This traceback is masking the real error. Try to run (from command-line) `bzr info bzr://ubuntu-srv-1.kzo-corp.com/svr/bzr/test.bazaar/trunk` and check what it says. IIUC you should get the real error.

[Alexander Belchenko (bialix)]

Dear bzr devs. This error partially comes from bzrlib itself. The code which triggers the error is roughly as following:

                try:
                    model = mod_location.LocationModel(location,
                        self._do_view_action, self.dry_run)
                except errors.BzrError, ex:
                    self._show_unable_to_open_location(location, ex,
                        auto_virtual_repo_open)

On BzrError the next method is invoked:

    def _show_unable_to_open_location(self, location, ex,
        auto_virtual_repo_open):

And it tries to convert BzrError object `ex` to unicode and it fails:

            why = gettext(unicode(ex))

Can you suggest something here?

[jean-michel (jeanmichel-cazaux)]

Ok, after a bit more digging, I realised that I might do something wrong.

From the explorers (both) I was trying to open bzr://server.example.com/srv/bzr/bazaar.test...
When changing this to bzr+ssh://<email address hidden>/srv/bzr/bazaar.test it all worked as expected.

Is it the intended behaviour? I mean is it normal that you cannot access the smart server using bzr protocol only?

[jean-michel (jeanmichel-cazaux)]

Hi Alex, when typing bzr info bzr://ubuntu-srv-1.kzo-corp.com/svr/bzr/bazaar.test in bzr command line on my windows dektop, I get:

C:\Program Files (x86)\Bazaar>bzr info bzr://ubuntu-srv-1.kzo-corp.com/svr/bzr/bazaar.test
bzr: ERROR: Connection error: failed to connect to ubuntu-srv-1.kzo-corp.com:4155: Aucune connexion nÆa pu Ûtre Útablie car lÆordinateur cible lÆa expressÚment refusÚe.

The "strange" characters that do replace French accents do indeed point at a translation issue.

[Alexander Belchenko (bialix)]

jean-michel пишет:
> Ok, after a bit more digging, I realised that I might do something
> wrong.
>
>>From the explorers (both) I was trying to open bzr://server.example.com/srv/bzr/bazaar.test...
> When changing this to bzr+ssh://<email address hidden>/srv/bzr/bazaar.test it all worked as expected.
>
> Is it the intended behaviour? I mean is it normal that you cannot access
> the smart server using bzr protocol only?

bzr:// protocol requires running server launched with command `bzr
serve` and it works via TCP (without any authorization).
bzr+ssh:// is automatically spawning `bzr serve` via ssh tunnel and it
uses ssh authorization. Please, read the corresponding part of User
Manual re different types of bzr smart server to understand it better.

--
All the dude wanted was his rug back

[jean-michel (jeanmichel-cazaux)]

Many thanks for that Alex, I will have a better look at the documentation, I am still evaluationg this thing.

Nonetheless, IMHO, parsing a unicode error message should not error out, should it?

Just my 2 (French) cents ;).

Thanks.
JM

PS: the dude was relly fond of his rug indeed.

[Alexander Belchenko (bialix)]

jean-michel пишет:
> Nonetheless, IMHO, parsing a unicode error message should not error out,
> should it?

The problem here IIUIC occurs because underlying OS gives us non-unicode
error message but localized (French) in latin-1 (most likely) encoding
or something similar. And internally we're trying to convert ot to
unicode without explicitly telling the original encoding. And Python
unicode library does not like that. If you're familiar with Python and
unicode issues then you should understand what I'm talking about.

Anyway, Bazaar Explorer should be more robust against this. And some
guard code should be present inside bzrlib, IMO.

[Martin Pool (mbp)]

hi, bialix,

There are some other bugs <https://bugs.launchpad.net/bzr/+bugs?field.searchtext=unicode+error> and eg <https://bugs.launchpad.net/bzr/+bug/404740> about handling unicode while formatting errors.

The general pattern of converting things to Unicode on the way in to the program seems like the way to go here. If Python or the OS given us an 8-bit encoded filename or error message in the original exception, we can try to decode it, bearing in mind the encoding may vary per error.

If we don't manage to decode it and we have unknown-encoding bytes floating around we ought to still try to print _something_ even if it has unreadable substrings; unfortunately that's going to be pretty bad for non-latin-based languages.

[Saša Janiška (gour)]

Hello,

I encountered this problem today while adding files to the fresh repo.

My report is attached to: https://bugs.launchpad.net/bzr-diffstat/+bug/56680 and similar/same thing is at https://bugs.launchpad.net/bzr-submit/+bug/67973

Can someone mark those bugs as duplicates and, pls. fix it before 2.4.

Sincerely,
Gour

[Saša Janiška (gour)]

Let me add that my 'project' is book which was written using LyX/LaTeX using latin-2 - it was in '99 when utf-8 was not so prevalent, but we still want to have it under DVCS (bzr).

Sincerely,
Gour

[Andrew Bennetts (spiv)]

Gour: I don't think those are the same as this bug. It appears the traceback you report at <https://bugs.launchpad.net/bzr-diffstat/+bug/56680/comments/9> is bug 715547. Please add any further comments there, thanks.

[Saša Janiška (gour)]

@Andrew: Thank you and excuse me for the noise. :-(