Broken unicode e-mails corrupt mailman's held message queue

[Lauri Ojansivu]

Dear Maintainers,

there's a bug in the Mailman3 package that leads to a corrupted held
messages queue. The problem manifests itself by a 500 server error when
the list administrator tries to access the respective list's "held
messages" panel in Postorius. Expected behaviour is that the held
messages queue is shown.

Details of the problem can be found in this thread:
https://<email address hidden>/thread/QWKXWB3D6GV7CACNH2I7A3JCM7OUHR3G/

Summary: If Mailman is served via Postfix, and Postfix has set the
"smtputf8_enable" configuration option set to "yes" (which as of Postfix
3.0.0 is the default value), then e-mail messages with broken Unicode
content are changed to include the U+FFFD REPLACEMENT CHARACTER. Such
broken messages are usually spam messages from non-members, and thus end
up in the held messages queue. There the U+FFFD character hits a bug in
Mailman3's e-mail processor, causing it to crash with a
UnicodeEncodeError, thus leading to the 500 server error when accessing
the held messages queue.

Once the corrupt message is in the held messages queue, there's no easy
way to remove it from it again. Remedies can be found in the linked
mailing list thread.

The problem is known upstream and has been fixed:

https://gitlab.com/mailman/mailman/issues/441
https://gitlab.com/mailman/mailman/merge_requests/350

It's an underlying problem in Python itself, reported here:
https://bugs.python.org/issue32330

For the record, this is the exception found in mailman.log:

Dec 09 13:21:04 2019 (3090) REST request handler error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/wsgiref/handlers.py", line 137, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/lib/python3/dist-packages/mailman/database/transaction.py", line 50, in
wrapper
    rtn = function(*args, **kws)
  File "/usr/lib/python3/dist-packages/mailman/rest/wsgiapp.py", line 214, in __call__
    return super().__call__(environ, start_response)
  File "falcon/api.py", line 215, in falcon.api.API.__call__ (falcon/api.c:2872)
  File "falcon/api.py", line 189, in falcon.api.API.__call__ (falcon/api.c:2419)
  File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 167, in
on_get
    resource = self._make_collection(request)
  File "/usr/lib/python3/dist-packages/mailman/rest/helpers.py", line 159, in
_make_collection
    for resource in collection]
  File "/usr/lib/python3/dist-packages/mailman/rest/helpers.py", line 159, in
<listcomp>
    for resource in collection]
  File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 157, in
_resource_as_dict
    resource = self._make_resource(request.id)
  File "/usr/lib/python3/dist-packages/mailman/rest/post_moderation.py", line 78, in
_make_resource
    resource['msg'] = msg.as_string()
  File "/usr/lib/python3.6/email/message.py", line 158, in as_string
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.6/email/generator.py", line 116, in flatten
    self._write(msg)
  File "/usr/lib/python3.6/email/generator.py", line 181, in _write
    self._dispatch(msg)
  File "/usr/lib/python3.6/email/generator.py", line 214, in _dispatch
    meth(msg)
  File "/usr/lib/python3.6/email/generator.py", line 243, in _handle_text
    msg.set_payload(payload, charset)
  File "/usr/lib/python3.6/email/message.py", line 315, in set_payload
    payload = payload.encode(charset.output_charset)
UnicodeEncodeError: 'ascii' codec can't encode character '\ufffd' in position 269: ordinal
not in range(128)

Please backport this fix. A corrupted held messages queue makes the
moderation of e-mails impossible.

Package mailman3 version: 3.1.1-9

Best Regards,
Lauri Ojansivu

[Launchpad Janitor (janitor)]

Status changed to 'Confirmed' because the bug affects multiple users.

[Haw Loeung (hloeung)]

We're carrying these patches against 3.1.1-9 (Bionic).

[Haw Loeung (hloeung)]

Also this.

[Ubuntu Foundations Team Bug Bot (crichton)]

The attachment "mailman_core_post_moderation.patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]