bzr smart server can't handle UTF-8 user names, gives UnknownErrorFromSmartServer

[Jan D.]

I have an ä (LATIN SMALL LETTER A WITH DIAERESIS) in my name, so it is the GECOS-field on savannah.gnu.org,
UTF-8 decoded.
When I tried to commit to bzr+ssh://<email address hidden>/emacs/emacs-23 (bound branch), I get an error back that says
  Unable to determine your name. Please use "bzr whoami" to set it (se log below).

I have used bzr whoami so the message is at least incorrect. Changing my name in savannah so that ä isn't in the GECOS-filed fixes the problem.
Log follows:

Sön 2010-10-10 14:51:11 +0200
0.044 bazaar version: 2.2.1
0.044 bzr arguments: [u'commit', u'-m', u"Bug#7150: Distinguishing between left
 and right Alt keys on NextStep/OSX.\n\n* lisp/cus-start.el (all): ns-right-alte
rnate-modifier is new.\n\n* lisp/term/ns-win.el (ns-right-alternate-modifier): N
ew defvar.\n(ns-right-option-modifier): New alias for ns-right-alternate-modifie
r.\n(mac-right-option-modifier): New alias for ns-right-option-modifier.\n\n* sr
c/nsterm.m (Qleft): Declare.\n(ns_right_alternate_modifier): New variable\n(NSRi
ghtAlternateKeyMask): New define.\n(EV_MODIFIERS): Parse NSRightAlternateKeyMask
 if\nns_right_alternate_modifier isn't Qleft.\n(keyDown): If ns_right_alternate_
modifier isn't Qleft, use it\nas emacs modifier for NSRightAlternateKeyMask.\n(s
yms_of_nsterm): DEFVAR_LISP ns-right-alternate-modifier.\n", u'etc/NEWS', u'lisp
/ChangeLog', u'lisp/cus-start.el', u'lisp/term/ns-win.el', u'src/ChangeLog', u's
rc/nsterm.m']
0.058 looking for plugins in /Users/jhd/.bazaar/plugins
0.058 looking for plugins in /Library/Python/2.6/site-packages/bzrlib/plugins
0.164 encoding stdout as osutils.get_user_encoding() 'UTF-8'
0.207 opening working tree '/Users/jhd/src/emacs/emacs-23'
0.599 bzr-svn: using Subversion 1.6.5 ()
0.643 ssh implementation is OpenSSH
4.522 preparing to commit
6.574 Traceback (most recent call last):
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 912, in exception_to_return_code
    return the_callable(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 1112, in run_bzr
    ret = run(*run_argv)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 690, in run_argv_aliases
    return self.run(**all_cmd_args)
  File "/Library/Python/2.6/site-packages/bzrlib/commands.py", line 705, in run
    return self._operation.run_simple(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 135, in run_simple
    self.cleanups, self.func, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/builtins.py", line 3200, in run
    exclude=safe_relpath_files(tree, exclude))
  File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/workingtree_4.py", line 197, in commit
    result = WorkingTree3.commit(self, message, revprops, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/decorators.py", line 194, in write_locked
    result = unbound(self, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/mutabletree.py", line 201, in commit
    *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/commit.py", line 286, in commit
    possible_master_transports=possible_master_transports)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 131, in run
    self.cleanups, self.func, self, *args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/cleanup.py", line 165, in _do_with_cleanups
    result = func(*args, **kwargs)
  File "/Library/Python/2.6/site-packages/bzrlib/commit.py", line 360, in _commit
    self._check_bound_branch(operation, possible_master_transports)
  File "/Library/Python/2.6/site-packages/bzrlib/commit.py", line 529, in _check_bound_branch
    self.master_branch.lock_write()
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 2442, in lock_write
    remote_tokens = self._remote_lock_write(token)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 2426, in _remote_lock_write
    repo_token or '', **err_context)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 57, in _call
    self._translate_error(err, **err_context)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 2296, in _translate_error
    self.repository._translate_error(err, branch=self, **context)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 819, in _translate_error
    self.bzrdir._translate_error(err, repository=self, **context)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 173, in _translate_error
    _translate_error(err, bzrdir=self, **context)
  File "/Library/Python/2.6/site-packages/bzrlib/remote.py", line 3003, in _translate_error
    raise errors.UnknownErrorFromSmartServer(err)
UnknownErrorFromSmartServer: Server sent an unexpected error: ('error', 'Unable to determine your name. Use "bzr whoami" to set it.')

6.574 return code 3

[Martin Pool (mbp)]

Thanks for the clear bug report.

I would speculate that Python may run on savannah with the locale set such that it expects gecos to be in ascii. You would think that LANG= whatever gets passed across ssh, but perhaps not. If you can get the server-side traceback, which should be in .bzr.log there, that could help.

It's also a bit questionable whether we actually need to read the gecos field on the server. This may be related to bzr trying to use your server-side name as information in the lock file.

[Jan D. (jan-h-d)]

The savannah admins say that there is no locale, the locale package is not installed.
Furthermore, they have disabled log files. Thread here: http://lists.gnu.org/archive/html/savannah-hackers/2010-10/msg00028.html.

[Martin Pool (mbp)]

Thanks for the link.

Sylvain says there

> I also think that the GECOS' encoding is generally independant from the user's current locale.

which is true, there's no guarantee that it is the same, but I'm not
aware of anything else that would tell us what it is. We can assume
it's ascii, assume it's utf-8 (perhaps most likely in 2010) or assume
it's the user's locale.

--
Martin

[Jan D. (jan-h-d)]

If the only purpose is to put information in a lock file, maybe just skipping non-ascii characters or treat is as a binary string is enough? Is this string you put there interpreted as a name by anybody else?

[Sylvain Beucler (beuc)]

I just got tricked into checking this issue again, here's the backtrace.
Version is http://packages.debian.org/lenny-backports/bzr - 2.0.3-1~bpo50+1

this is a debug log for diagnosing/reporting problems in bzr
you can delete or truncate this file, or include sections in
bug reports to https://bugs.launchpad.net/bzr/+filebug

Mon 2010-11-01 20:17:26 +0000
0.127 bzr arguments: [u'--no-plugins', u'serve', u'--inet', u'--directory=/srv/bzr', u'--allow-writes']
0.262 encoding stdout as osutils.get_user_encoding() 'ANSI_X3.4-1968'
0.367 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 893, in accept_bytes
    _StatefulDecoder.accept_bytes(self, bytes)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 388, in accept_bytes
    self.state_accept()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 999, in _state_accept_expecting_message_part
    self.done()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 1037, in done
    raise errors.SmartMessageHandlerError(sys.exc_info())
SmartMessageHandlerError: The message handler raised an exception:
Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/protocol.py", line 1035, in done
    self.message_handler.end_received()
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/message.py", line 164, in end_received
    "Complete conventional request was received, but request "
SmartProtocolError: Generic bzr smart protocol error: Complete conventional request was received, but request handler has not finished reading.

1.970 Traceback (most recent call last):
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/request.py", line 326, in _call_converting_errors
    return callable(*args, **kwargs)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/request.py", line 132, in execute
    return self.do(*args)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/branch.py", line 49, in do
    return self.do_with_branch(branch, *args)
  File "/usr/lib/python2.5/site-packages/bzrlib/smart/branch.py", line 297, in do_with_branch
    branch_token = branch.lock_write(token=branch_token)
  File "/usr/lib/python2.5/site-packages/bzrlib/branch.py", line 2135, in lock_write
    return self.control_files.lock_write(token=token)
  File "/usr/lib/python2.5/site-packages/bzrlib/lockable_files.py", line 194, in lock_write
    token_from_lock = self._lock.lock_write(token=token)
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 579, in lock_write
    return self.wait_lock()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 503, in wait_lock
    return self.attempt_lock()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 464, in attempt_lock
    result = self._attempt_lock()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 222, in _attempt_lock
    tmpname = self._create_pending_dir()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 280, in _create_pending_dir
    info_bytes = self._prepare_info()
  File "/usr/lib/python2.5/site-packages/bzrlib/lockdir.py", line 439, in _prepare...

[Vincent Ladeuil (vila)]

From the code point of view, the user name is decoded with the so-called user_encoding which should apply to the user *input*, so arguably we shouldn't use it for a file content... even if this sounds like the best bet in most of the cases.

One workaround that doesn't require any bzr change is to use user names that can be decoded with the declared encoding which is: 'ANSI_X3.4-1968'. I.e. ascii.

Note that you're using bzr-2.0.3 and that most recent versions establish the user name in a slightly different way (and it's unclear to me how these versions will behave here :-/).

[Martin Pool (mbp)]

vila mentioned this in https://code.launchpad.net/~mbp/bzr/whoami/+merge/55679 but because the relevant code there is not new, I doubt that patch will fix it.

[Martin Pool (mbp)]

I've just tested my fix <https://code.launchpad.net/~mbp/bzr/whoami/+merge/55679> with my own gecos entry set to include unicode non-latin characters, and it seems to work ok. If not, please let us know. That fix will be in the next 2.3 release.

[Martin Pitt (pitti)]

Accepted bzr into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Clint Byrum (clint-fewbar)]

Hello Jan, or anyone else affected,

Accepted bzr into natty-proposed, the package will build now and be available in a few hours. Please test and give feedback here. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance!

[Jelmer Vernooij (jelmer)]

Verified by running the bzr testsuite from the package in a clean natty install.

[Launchpad Janitor (janitor)]

This bug was fixed in the package bzr - 2.3.4-0ubuntu1

---------------
bzr (2.3.4-0ubuntu1) natty-proposed; urgency=low

  * New upstream release.
   + Fix bzr version number in deprecation warnings. LP: #794960
   + Prevent write attemps on remote branch during "bzr up". LP: #786980
   + Fix conflict handling when two trees involved in a merge have different
     root ids. LP: #805809

bzr (2.3.3-0ubuntu1) natty-proposed; urgency=low

  * New upstream release.
   + Fixes deprecation warning on newer versions of Python. LP: #760435
   + Stops 'bzr push' from copying entire repository if a .bzr directory is
     present without a branch. LP: #465517
   + Fixes undefined local variable error when waiting for lock. LP: #733136
   + Fixes lock contention issues pushing to a bound branch. LP: #733350
   + Transfers less data creating a new stacked branch. LP: #737234
   + Several fixes to the test suite, making it more robust. LP: #654733,
      LP: #751824
   + 'bzr merge --pull --preview' actually shows a preview rather than
     actually merging. LP: #760152
   + bzr smart server now supports UTF-8 user names. LP: #659763
   + user identity can now be set based on username and /etc/mailname, not
     requiring it to be set manually. LP: #616878
   + stacking is now fully transitive. LP: #715000
   + makes in-terminal crash report of plugins much shorter. LP: #716389
 -- Jelmer Vernooij <email address hidden> Thu, 14 Jul 2011 21:12:58 +0200