SchoolTool hangs when username includes german Umlaut

[Martin Schimmels]

Users that have non-ascii symbols in username cannot login, they get a traceback:

  UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 12: ordinal not in range(128)
Zurückverfolgung

  File "/usr/lib/python2.5/site-packages/zope/publisher/publish.py", line 130, in publish
  obj = request.traverse(obj)
File "/usr/lib/python2.5/site-packages/zope/publisher/browser.py", line 513, in traverse
  ob = super(BrowserRequest, self).traverse(obj)
File "/usr/lib/python2.5/site-packages/zope/publisher/http.py", line 464, in traverse
  ob = super(HTTPRequest, self).traverse(obj)
File "/usr/lib/python2.5/site-packages/zope/publisher/base.py", line 252, in traverse
  publication.callTraversalHooks(self, obj)
File "/usr/lib/python2.5/site-packages/zope/app/publication/zopepublication.py", line 129, in callTraversalHooks
  self._maybePlacefullyAuthenticate(request, ob)
File "/usr/lib/python2.5/site-packages/zope/app/publication/zopepublication.py", line 123, in _maybePlacefullyAuthenticate
  request.setPrincipal(principal)
File "/usr/lib/python2.5/site-packages/zope/publisher/http.py", line 516, in setPrincipal
  message = logging_info.getLogMessage()
File "/usr/lib/python2.5/site-packages/zope/app/security/principallogging.py", line 29, in getLogMessage
  return str(self.principal.id)

[Justas Sadzevičius (justas.sadzevicius)]

Thank you for the bug report.

We will prohibit usernames with non-ascii codec in the next release.

[Tom Hoffman (tom-hoffman)]

So this is a Zope thing, not a SchoolTool thing?

[Douglas Cerna (replaceafill)]

When SchoolTool creates the Principal object in schooltool.app.security.PersonContainerAuthenticationPlugin.getPrincipal it uses:

principal = Principal(id, person.title, person=ProxyFactory(person))

where id is a unicode like u'sb.person.Zur\xfcckverfolgung'. The Principal class tries to encode this using ASCII and it fails. If I change the previous line to:

principal = Principal(id.encode("punycode"), person.title, person=ProxyFactory(person))

I can login as Zurückverfolgung. I don't know the repercussions on functionality if you change the principal id like this, but I like unicode usernames :)

[Tom Hoffman (tom-hoffman)]

If we can test and implement this with a reasonably amount of work, we should. If SchoolTool is successful we'll have a *lot* of schools with entirely non-ASCII character sets. We put a lot of work into making SchoolTool work under those conditions, so it would be a shame to drop the ball on something as basic as person ID's.

[Gediminas Paulauskas (menesis)]

For 1.4.0 a check was added that prevents adding users with non-ascii usernames.

Existing users will not be able to login as before, that will have to be fixed later.

[Gediminas Paulauskas (menesis)]

We might have issues with unicode usernames, I have fixed two such bugs (Bug #495872 and http://bazaar.launchpad.net/~schooltool-owners/schooltool.gradebook/trunk/revision/163).

But this error, that prevents such users from loggin in, is a bug #131460 in zope.publisher that will be fixed in 3.12.4 (unreleased). When that is released, use it in development schooltool, remove the usernames restriction, and look for more bugs.

Douglas' comment #3 suggests storing an encoded principal id. The fix in zope.publisher is similar, except that it uses principal.id.encode('ascii', 'backslashreplace') instead of id.encode("punycode") and does that only for log messages, id is stored as unicode.

[Justas Sadzevičius (justas.sadzevicius)]

One more traceback, when a student ąžuolas looks at the Manage view (Unauthorized).

2010-09-21 16:10:44,062 Exception while rendering view on exception
Traceback (most recent call last):
  File "/home/justas/.buildout/eggs/zope.app.publication-3.11.1-py2.6.egg/zope/app/publication/zopepublication.py", line 379, in handleException
    transaction.commit()
  File "/home/justas/.buildout/eggs/transaction-1.0.0-py2.6.egg/transaction/_manager.py", line 93, in commit
    return self.get().commit()
  File "/home/justas/.buildout/eggs/transaction-1.0.0-py2.6.egg/transaction/_transaction.py", line 322, in commit
    self._commitResources()
  File "/home/justas/.buildout/eggs/transaction-1.0.0-py2.6.egg/transaction/_transaction.py", line 419, in _commitResources
    rm.tpc_vote(self)
  File "/home/justas/.buildout/eggs/ZODB3-3.9.4-py2.6-linux-i686.egg/ZODB/Connection.py", line 767, in tpc_vote
    s = vote(transaction)
  File "/home/justas/.buildout/eggs/ZODB3-3.9.4-py2.6-linux-i686.egg/ZODB/FileStorage/FileStorage.py", line 724, in tpc_vote
    self._file.write(h.asString())
  File "/home/justas/.buildout/eggs/ZODB3-3.9.4-py2.6-linux-i686.egg/ZODB/FileStorage/format.py", line 278, in asString
    return "".join(map(str, [s, self.user, self.descr, self.ext]))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 12-13: ordinal not in range(128)

[Gediminas Paulauskas (menesis)]

The same exception as comment#7 if the user saves anything, e.g. calendar preferences.

I would say this is a bug in ZODB3, the asString function (last line of traceback).

[Gediminas Paulauskas (menesis)]

I have reported this ZODB3 issue as Bug #646019.

[Gediminas Paulauskas (menesis)]

We have to remove these ascii checks after zope.app.publication Bug #646019 is fixed. Then test again that there are no new bugs related to unicode usernames.