Unicode errors in mailman3

Bug #1130957 reported by Aurélien Bompard on 2013-02-20
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
GNU Mailman
High
Barry Warsaw

Bug Description

This is with mailman3. I'm getting the following unicode errors:

Feb 20 23:07:43 2013 (13496) Uncaught runner exception: Expected unicode, found <type 'str'>: '<email address hidden>'
Feb 20 23:07:43 2013 (13496) Traceback (most recent call last):
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration
    self._process_one_file(msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/runners/outgoing.py", line 93, in _dispose
    self._func(mlist, msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/mta/deliver.py", line 93, in deliver
    refused = agent.deliver(mlist, msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/mta/base.py", line 167, in deliver
    member = mlist.members.get_member(recipient)
  File "/srv/dev/mailman/mailman/src/mailman/database/transaction.py", line 78, in wrapper
    return function(args[0], config.db.store, *args[1:], **kws)
  File "/srv/dev/mailman/mailman/src/mailman/model/roster.py", line 111, in get_member
    Address.email == address,
  File "/srv/dev/mailman/venv/lib/python2.7/site-packages/storm-0.19-py2.7-linux-x86_64.egg/storm/expr.py", line 412, in __eq__
    other = getattr(self, "variable_factory", Variable)(value=other)
  File "/srv/dev/mailman/venv/lib/python2.7/site-packages/storm-0.19-py2.7-linux-x86_64.egg/storm/variables.py", line 396, in parse_set
    % (type(value), value))
TypeError: Expected unicode, found <type 'str'>: '<email address hidden>'
Feb 20 23:07:43 2013 (13496) SHUNTING: 1361401663.441109+267302c50d71d512974fe5caa7aa0214eab699f6

-----------------

Feb 20 22:20:45 2013 (13337) Uncaught runner exception: decoding Unicode is not supported
Feb 20 22:20:45 2013 (13337) Traceback (most recent call last):
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration
    self._process_one_file(msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 344, in _dispose
    rfc1153_digest.add_message(message, count)
  File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 269, in add_message
    payload = unicode(payload, 'us-ascii', 'replace')
TypeError: decoding Unicode is not supported
Feb 20 22:20:45 2013 (13337) SHUNTING: 1361398845.847068+d12dc2ef44dcca60ade2fc94f6f830bd09a6b912
Feb 20 22:20:52 2013 (13337) Uncaught runner exception: decoding Unicode is not supported
Feb 20 22:20:52 2013 (13337) Traceback (most recent call last):
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 145, in _one_iteration
    self._process_one_file(msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/core/runner.py", line 228, in _process_one_file
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 344, in _dispose
    rfc1153_digest.add_message(message, count)
  File "/srv/dev/mailman/mailman/src/mailman/runners/digest.py", line 269, in add_message
    payload = unicode(payload, 'us-ascii', 'replace')
TypeError: decoding Unicode is not supported
Feb 20 22:20:52 2013 (13337) SHUNTING: 1361398852.901291+bc2f1ab7dad828020238e06815fd2b4adab0fe61

The attached patch fixes both tracebacks.

Related branches

Aurélien Bompard (abompard) wrote :
Barry Warsaw (barry) wrote :

Can you provide a reproducible recipe, or test case? Is this caused by a member with non-ASCII in their name? Or a message with non-ASCII in the From field? Any more information about how to reproduce this will really help (e.g. to create a test case).

Aurélien Bompard (abompard) wrote :

I'm trying to write a testcase for the second set of tracebacks, but I'm having a hard time understanding where the message that causes it comes from. When I open the pickle file I get:
'From nobody Sat Mar 23 09:42:34 2013\n\n'
This goes through the digest runner and tracebacks at line 269 with "TypeError: decoding Unicode is not supported"

Do you know where such a message can come from ? Are you OK with just writing a test case with this string as an entry message ?

Aurélien Bompard (abompard) wrote :

I'm still trying to figure out the unicode errors in the digest runner, but something happens that I don't understand.
In the add_message method, around line 264, the msg.as_string() call returns an str instance, but the msg.as_string().split("\n\n") call returns a list of unicode instances. This makes no sense to me.
Could it come from the unicode-related overrides in mailman.email.message ? Could it be a bug in the python interpreter ? I'm using python 2.7.3. I can't find a way to reproduce it outside mailman.

Aurélien Bompard (abompard) wrote :

OK, thanks to Toshio, I *finally* know what's going on here. Because of the import of "unicode_literals" from __future__ at the top of the file, the call to .split("\n\n") is actually interpreted as .split(u"\n\n"), thus the result is a list of unicode strings, which of course can't be decoded further in the following lines. To keep string literals, the string should be prefixed with "b", here's the patch:

=== modified file 'src/mailman/runners/digest.py'
--- src/mailman/runners/digest.py 2013-01-01 14:05:42 +0000
+++ src/mailman/runners/digest.py 2013-04-11 15:52:53 +0000
@@ -260,7 +260,7 @@
         # Add the payload. If the decoded payload is empty, this may be a
         # multipart message. In that case, just stringify it.
         payload = msg.get_payload(decode=True)
- payload = (payload if payload else msg.as_string().split('\n\n', 1)[1])
+ payload = (payload if payload else msg.as_string().split(b'\n\n', 1)[1])
         try:
             charset = msg.get_content_charset('us-ascii')
             payload = unicode(payload, charset, 'replace')

That's one of the errors I reported in this but, I'll investigate the other one now.

Aurélien Bompard (abompard) wrote :

OK, I found out where the second error comes from. When there is no administrator available, the message is sent to the site_owner from the configuration file, but this entry is not converted to unicode before being set in the msgdata dict.
Thus, when it's added to the database, Storm refuses it. The following patch fixes this bug:

=== modified file 'src/mailman/handlers/owner_recipients.py'
--- src/mailman/handlers/owner_recipients.py 2013-01-01 14:05:42 +0000
+++ src/mailman/handlers/owner_recipients.py 2013-04-12 16:47:29 +0000
@@ -55,7 +55,7 @@
         # To prevent -owner messages from going into a black hole, if there
         # are no administrators available, the message goes to the site owner.
         if len(recipients) == 0:
- msgdata['recipients'] = set((config.mailman.site_owner,))
+ msgdata['recipients'] = set((unicode(config.mailman.site_owner),))
         else:
             msgdata['recipients'] = recipients
         # Don't decorate these messages with the header/footers. Eventually

Tim Marx (tim-marx93) wrote :

I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same problem as described above.
Luckily I found this bug report and could manage to fix it with the patches above.

May I ask anybody (with more bazar-knowledge than me) to merge this two patches into the mailman 3 branch?
Thank you!

On Apr 30, 2014, at 09:38 PM, Tim Marx wrote:

>I'm using mailman 3.0.0b4 (with mailman-bundler) and ran into the same
>problem as described above. Luckily I found this bug report and could manage
>to fix it with the patches above.
>
>May I ask anybody (with more bazar-knowledge than me) to merge this two
>patches into the mailman 3 branch? Thank you!

I think we just need some test cases. Tim, can you provide any details on how
you it the bug?

Aurélien Bompard (abompard) wrote :

OK, I've written the unit tests and pushed them with the fixes in my dedicated branch: lp:~abompard/mailman/bug-1130957. The merge proposal has just been sent.

Barry Warsaw (barry) on 2014-11-29
Changed in mailman:
milestone: none → 3.0.0b5
assignee: nobody → Barry Warsaw (barry)
importance: Undecided → High
status: New → Fix Committed
Barry Warsaw (barry) on 2014-12-30
Changed in mailman:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers