UnicodeEncodeError on email-in repoze.postoffice

[Paul Everitt]

The error log:

https://karl.soros.org/error_monitor_subsystem.html?subsystem=mailin

...is full of entries such as:

Sat Sep 11 21:47:10 2010 Error in daemon process Traceback (most recent call last): File "/opt/karl/osi/3.26-0/src/karl/karl/scripting.py", line 74, in run_daemon func() File "/opt/karl/osi/3.26-0/src/karl/karl/scripts/mailin.py", line 197, in run runner() File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 199, in __call__ if self.handle_message(message): File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 233, in handle_message error_msg = self.quarantine_message(message) File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 286, in quarantine_message self.queue.quarantine(message, error, mailer.send, from_email) File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/queue.py", line 192, in quarantine notice['To'] = message['From'] File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py", line 25, in __setitem__ StdlibMessage.__setitem__(self, name, encode_header(name, value)) File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py", line 71, in encode_header str(Header(user)), addr.encode('ascii') UnicodeEncodeError: 'ascii' codec can't encode character u'\u0432' in position 0: ordinal not in range(128)

[Paul Everitt (paul-agendaless)]

Probably one of the higher priority things for us to fix.

[Chris Rossi (chris-archimedeanco)]

OSI seems to be the target of a new crop of spambots which have no regards for mime message specification with regards to unicode in headers. I have removed the offending spam messages from OSI's queue so that the mailin can once again, flow, so they are unstuck for now. I will add some handling to postoffice to deal with these particular non-standards compliant spambots. Because I can't find the spammers and punch them.

[Paul Everitt (paul-agendaless)]

I also wish you could punch them.

Is this something we need to put on the SFU plate to get better MTA or Postoffice blocking?

--Paul

On Sep 13, 2010, at 4:00 PM, Chris Rossi wrote:

> OSI seems to be the target of a new crop of spambots which have no
> regards for mime message specification with regards to unicode in
> headers. I have removed the offending spam messages from OSI's queue so
> that the mailin can once again, flow, so they are unstuck for now. I
> will add some handling to postoffice to deal with these particular non-
> standards compliant spambots. Because I can't find the spammers and
> punch them.
>
> --
> UnicodeEncodeError on email-in repoze.postoffice
> https://bugs.launchpad.net/bugs/637484
> You received this bug notification because you are a direct subscriber
> of the bug.
>
> Status in KARL3: Confirmed
>
> Bug description:
> The error log:
>
> https://karl.soros.org/error_monitor_subsystem.html?subsystem=mailin
>
> ...is full of entries such as:
>
> Sat Sep 11 21:47:10 2010 Error in daemon process Traceback (most recent call last): File "/opt/karl/osi/3.26-0/src/karl/karl/scripting.py", line 74, in run_daemon func() File "/opt/karl/osi/3.26-0/src/karl/karl/scripts/mailin.py", line 197, in run runner() File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 199, in __call__ if self.handle_message(message): File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 233, in handle_message error_msg = self.quarantine_message(message) File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 286, in quarantine_message self.queue.quarantine(message, error, mailer.send, from_email) File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/queue.py", line 192, in quarantine notice['To'] = message['From'] File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py", line 25, in __setitem__ StdlibMessage.__setitem__(self, name, encode_header(name, value)) File "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py", line 71, in encode_header str(Header(user)), addr.encode('ascii') UnicodeEncodeError: 'ascii' codec can't encode character u'\u0432' in position 0: ordinal not in range(128)
>
> To unsubscribe from this bug, go to:
> https://bugs.launchpad.net/karl3/+bug/637484/+subscribe

[Chris Rossi (chris-archimedeanco)]

The fix for these particular problems is released in repoze.postoffice 0.6. All partner sites will get the fix on the next update. How imperative it is to update quickly kind of depends on how often we get bombarded by these spambots. Note that because the fix is in an external library, no new release of Karl or any customization packages is necessary. Really just buildout needs to be rerun to get the newer egg. So Jim B could theoretically run updates using the same tags as his last update if we need to get just this change.

Paul, as far as your question is concerned, anything to prevent spam from getting to repoze.postoffice is potentially beneficial. The vast majority of cases that I've seen of broken mail-in have been due to incoming spam. Spam messages are the most likely to not conform to mime standards or use headers in unexpected ways, and are therefore the most likely to break mail-in.

[Paul Everitt (paul-agendaless)]

JimP/Chris, I believe this one can now be marked as Fix Released, right?

[Chris Rossi (chris-archimedeanco)]

I believe so. There is a new error having to do with someone sending email
in a character encoding we don't know, but that should be a new ticket.

Chris

On Tue, Oct 12, 2010 at 6:37 AM, Paul Everitt <email address hidden> wrote:

> JimP/Chris, I believe this one can now be marked as Fix Released, right?
>
> --
> UnicodeEncodeError on email-in repoze.postoffice
> https://bugs.launchpad.net/bugs/637484
> You received this bug notification because you are a bug assignee.
>
> Status in KARL3: Fix Committed
>
> Bug description:
> The error log:
>
> https://karl.soros.org/error_monitor_subsystem.html?subsystem=mailin
>
> ...is full of entries such as:
>
> Sat Sep 11 21:47:10 2010 Error in daemon process Traceback (most recent
> call last): File "/opt/karl/osi/3.26-0/src/karl/karl/scripting.py", line 74,
> in run_daemon func() File
> "/opt/karl/osi/3.26-0/src/karl/karl/scripts/mailin.py", line 197, in run
> runner() File "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line
> 199, in __call__ if self.handle_message(message): File
> "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 233, in
> handle_message error_msg = self.quarantine_message(message) File
> "/opt/karl/osi/3.26-0/src/karl/karl/utilities/mailin.py", line 286, in
> quarantine_message self.queue.quarantine(message, error, mailer.send,
> from_email) File
> "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/queue.py",
> line 192, in quarantine notice['To'] = message['From'] File
> "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py",
> line 25, in __setitem__ StdlibMessage.__setitem__(self, name,
> encode_header(name, value)) File
> "/opt/karl/osi/3.26-0/eggs/repoze.postoffice-0.5-py2.5.egg/repoze/postoffice/message.py",
> line 71, in encode_header str(Header(user)), addr.encode('ascii')
> UnicodeEncodeError: 'ascii' codec can't encode character u'\u0432' in
> position 0: ordinal not in range(128)
>
>
>

[Paul Everitt (paul-agendaless)]

Jim, can you confirm that this is wrapped up? I agree with Chris that the other error shouldn't count toward this error, but instead is a new case.

[JimPGlenn (jpglenn09)]

fixed