Japanese language prevents user from unsubscribing

Bug #1921682 reported by Tomas Korbar on 2021-03-29
This bug affects 1 person
Affects Status Importance Assigned to Milestone
GNU Mailman
Mark Sapiro

Bug Description

When mailing list is configured with:

preferred_language = 'ja'
available_languages = ['en', 'ja']
send_goodbye_msg = 0

And mailman receives confirmation email for unsubscription with multipart message encoded in
different encoding than euc-jp, then mailmans command runner crashes with traceback:

Feb 10 18:08:00 2021 (1270) Uncaught runner exception: 'euc_jp' codec can't decode bytes in position 280-281: illegal multibyte sequence
Feb 10 18:08:00 2021 (1270) Traceback (most recent call last):
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 119, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 190, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 291, in _dispose
  File "/usr/lib/mailman/Mailman/Queue/CommandRunner.py", line 208, in send_response
    results = MIMEText(NL.join(encoded_resp), _charset=charset)
  File "/usr/lib64/python2.7/email/mime/text.py", line 30, in __init__
    self.set_payload(_text, _charset)
  File "/usr/lib64/python2.7/email/message.py", line 226, in set_payload
  File "/usr/lib64/python2.7/email/message.py", line 264, in set_charset
    self._payload = charset.body_encode(self._payload)
  File "/usr/lib64/python2.7/email/charset.py", line 390, in body_encode
    s = self.convert(s)
  File "/usr/lib64/python2.7/email/charset.py", line 273, in convert
    return unicode(s, self.input_codec).encode(self.output_codec)
UnicodeDecodeError: 'euc_jp' codec can't decode bytes in position 280-281: illegal multibyte sequence

Here is an example of a mail which triggers this:

Date: Wed, 11 Nov 2020 14:13:16 +0900
Subject: Re: ###CONFIRM###
From: <testuser2@###SERVER###>
To: userlist-request@###SERVER###.com
MIME-Version: 1.0
Content-Type: multipart/alternative; boundary="000000000000663b1c05b3cdda78"

Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: base64



Also its maybe worth pointing out that setting RESPONSE_INCLUDE_LEVEL to <=1 mitigates this.

I've managed to fix this issue by decoding the message part with its original charset and then encoding it to preferred encoding of mailing list.
I'm attaching my patch. Could you please consider merging it?

Thanks for any help you can provide.

Related branches

Tomas Korbar (tkorbar) wrote :
Mark Sapiro (msapiro) wrote :

Thank you for the patch.

Changed in mailman:
assignee: nobody → Mark Sapiro (msapiro)
importance: Undecided → Medium
milestone: none → 2.1.35
status: New → Fix Committed
Mark Sapiro (msapiro) wrote :

The current fix is attached. I initially committed the change without the 'errors=' arguments in the unicode and encode even though I knew better, and it only took 42 minutes after I installed it on mail.python.org to throw a UnicodeEncodeError on a confirmation on an English language list with non-ascii in the body.

Tomas Korbar (tkorbar) wrote :

Ah, sorry about that.
Thanks for the new patch.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers