zope.sendmail: TypeError: character mapping must return integer, None or unicode

Reported by volker.jaenisch on 2010-06-22
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
zope.sendmail
Undecided
Martijn Pieters

Bug Description

Due to a change in python2.6 smtplib

[see http://bugs.python.org/issue8489]

it is no longer possible to send unicode login credential to the mailer,
since smtplib expect plain strings or utf-8 codet strings.

This simple patch solves the problem

zope.sendmail-3.7.1-py2.6.egg/zope/sendmail$ diff mailer.py mailer.py~
67c67
< connection.login(self.username.encode('utf-8'), self.password.encode('utf-8'))
---
> connection.login(self.username, self.password)

Best Regards

Volker Jaenisch

Christian Theune (ct-gocept) wrote :

Please do not file bugs with Zope 3 anymore, but assign them to the individual packages.

affects: zope3 → zope.sendmail

Sorry!

Christian Theune schrieb:
> Please do not file bugs with Zope 3 anymore, but assign them to the
> individual packages.
>
> ** Project changed: zope3 => zope.sendmail
>
Will not happen again.

Best Regards,

Volker

--
====================================================
   inqbus it-consulting +49 ( 341 ) 5643800
   Dr. Volker Jaenisch http://www.inqbus.de
   Herloßsohnstr. 12 0 4 1 5 5 Leipzig
   N O T - F Ä L L E +49 ( 170 ) 3113748
====================================================

Martijn Pieters (mjpieters) wrote :

The fact that smtplib accepted unicode before was more of an accident than a feature, something that changed with an optimisation in the hmac module, see http://bugs.python.org/issue5285.

Patch looks correct to me. However, can someone find out if UTF-8 makes any sense here. I see a Django newsletter product encode to utf8 here (see http://github.com/Fantomas42/emencia-django-newsletter/issues/issue/1 for example), and so does Trac (http://trac.edgewall.org/ticket/8083). I'd like to see some confirmation though that SMTP servers accept non-ascii characters in usernames or passwords in UTF-8, or if there are any other encodings that need to be supported.

When the answer is that only ASCII is supported, I'd say UTF-8 is just fine here as it protects the user that enters a username and password that won't work anyway will not get a strange encoding error, just failure to send email (as expected).

Martijn Pieters (mjpieters) wrote :

http://tools.ietf.org/html/draft-ietf-sasl-crammd5-10 applies here. It's a draft specification to update the CRAM MD5 SASL standard to define how to deal with non-ASCII usernames and passwords. It explicitly states usernames and passwords should be encoded to UTF-8.

The python HMAC library change only exposed the bug that smtplib accepted unicode strings for the username and password, and the CRAM MD5 mechanism is not the only authentication method. Although the other methods continue to work with unicode values, they are just as broken in practice. One of the other methods is PLAIN SASL (http://www.ietf.org/rfc/rfc4616.txt), and it already states the encoding used is UTF-8.

So, to summarize my findings: Sending in unencoded Unicode strings is broken for non-ASCII usernames and passwords already. Encoding them is the correct fix, and UTF-8 is the correct encoding for PLAIN, and for the proposed update for CRAM MD5. As any non-ASCII usernames and passwords are broken now, the patch won't make matters worse for those cases.

I'll apply the patch.

Martijn Pieters (mjpieters) wrote :

For the record, the 3rd and last supported SASL mechanism is LOGIN (http://sepp.oetiker.ch/sasl-2.1.19-ds/draft-murchison-sasl-login-00.txt) and it's specification also states that UTF-8 is the correct encoding.

Marking as assigned to me.

Changed in zope.sendmail:
assignee: nobody → Martijn Pieters (mjpieters)
status: New → Confirmed
status: Confirmed → In Progress
Martijn Pieters (mjpieters) wrote :

I fixed the problem slightly differently, to deal with the case where username and password are already encoded; utf-8 encoding is only applied when the parameter in question is a unicode string.

Fix committed in r117102 for trunk, r117103 for the 3.7 branch.

Changed in zope.sendmail:
status: In Progress → Fix Committed
Martijn Pieters (mjpieters) wrote :

Hanno released zope.sendmail 3.7.4 which includes this fix.

Changed in zope.sendmail:
status: Fix Committed → Fix Released

Perfect!

Thank you a lot.

Volker Jaenisch

Am 01.10.2010 11:39, schrieb Martijn Pieters:
> Hanno released zope.sendmail 3.7.4 which includes this fix.
>
> ** Changed in: zope.sendmail
> Status: Fix Committed => Fix Released
>
>

--
====================================================
   inqbus it-consulting +49 ( 341 ) 5643800
   Dr. Volker Jaenisch http://www.inqbus.de
   Herloßsohnstr. 12 0 4 1 5 5 Leipzig
   N O T - F Ä L L E +49 ( 170 ) 3113748
====================================================

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.