HTML email disappears

[Sjreeves]

In mailman-2.1.6rc2, messages sent in HTML are not
posted but simply disappear. If I replace the Messages.py
file with the one from 2.1.4, the problem goes away. I
found the following in the error log file, which seems to
point to this file:

May 04 15:16:59 2005 (23119) SHUNTING:
1115245017.9840169+c66ab2072c1f04236dd473ee99eb90
2f7aead680
May 04 15:39:16 2005 (23119) Uncaught runner exception:
unicode() argument 2 must be string, not None
May 04 15:39:16 2005 (23119) Traceback (most recent
call last):
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line
111, in _oneloop
    self._onefile(msg, msgdata)
  File "/usr/local/mailman/Mailman/Queue/Runner.py", line
167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)

File "/usr/local/mailman/Mailman/Queue/OutgoingRunner.py
", line 73, in _dispose
    self._func(mlist, msg, msgdata)

File "/usr/local/mailman/Mailman/Handlers/SMTPDirect.py",
line 152, in process
    deliveryfunc(mlist, msg, msgdata, envsender, refused,
conn)

File "/usr/local/mailman/Mailman/Handlers/SMTPDirect.py",
line 285, in verpdeliver
    Decorate.process(mlist, msgcopy, msgdata)
  File "/usr/local/mailman/Mailman/Handlers/Decorate.py",
line 88, in process
    mcset = msg.get_content_charset() or 'us-ascii'
  File "/usr/local/mailman/Mailman/Message.py", line 204, in
get_content_charset
    unicode('x', charset)
TypeError: unicode() argument 2 must be string, not None

[http://sourceforge.net/tracker/index.php?func=detail&aid=1195569&group_id=103&atid=100103]

[David Beecher (dbeecher)]

We are having a similar issue. Running version 2.1.9 on RHEL4. We do not lose every email that is HTML. It is random and so far appears to be affecting only 1 user (although I suspect it occasionally happens to others as well). The message is successfully getting to mailman but is "disappearing" and here' s what we're seeing in the log:

Apr 05 23:56:35 2009 (4051) Uncaught runner exception: Expected list, got <type 'str'>
Apr 05 23:56:35 2009 (4051) Traceback (most recent call last):
  File "/tekops/mailman2/Mailman/Queue/Runner.py", line 112, in _oneloop
    self._onefile(msg, msgdata)
  File "/tekops/mailman2/Mailman/Queue/Runner.py", line 170, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File "/tekops/mailman2/Mailman/Queue/IncomingRunner.py", line 130, in _dispose
    more = self._dopipeline(mlist, msg, msgdata, pipeline)
  File "/tekops/mailman2/Mailman/Queue/IncomingRunner.py", line 153, in _dopipeline
    sys.modules[modname].process(mlist, msg, msgdata)
  File "/tekops/mailman2/Mailman/Handlers/MimeDel.py", line 95, in process
    collapse_multipart_alternatives(msg)
  File "/tekops/mailman2/Mailman/Handlers/MimeDel.py", line 184, in collapse_multipart_alternatives
    firstalt = subpart.get_payload(0)
  File "/tekops/mailman2/pythonlib/email/Message.py", line 211, in get_payload
    raise TypeError, 'Expected list, got %s' % type(self._payload)
TypeError: Expected list, got <type 'str'>

Thanks,
David

[Mark Sapiro (msapiro)]

The original bug reported here was fixed in 2.1.6rc3.

The problem in David Beecher's comment appears to be due to a malformed message with a multipart/alternative part with no sub-parts. Possibly the original message had alternative parts that were removed prior to collapse_multipart_alternatives processing.

If possible, please attach or send me the .pck file from the shunt queue that was produced by this error.

[David Beecher (dbeecher)]

Hello Mark,
I'm emailing you the PCK file because I don't know if it contains 'sensitive' information. I ran strings on a few of the *.pck files I found in that directory and they had password data, etc.
Thanks,
David

[Mark Sapiro (msapiro)]

Thanks for emailing the shunted .pck file. The message in that file is defective. It sort of has the following MIME structure.

multipart/mixed
    text/plain
    multipart/alternative
        text/plain

The last text/plain part is an AVG anti-virus certification. There may have also been a text/html part that was already removed by content filtering before the message was shunted.

The problem is the multipart/alternative part specifies the same boundary string as the outer part so the message parses as

multipart/mixed
    text/plain
    multipart/alternative
    text/plain

with an empty multipart/alternative part. I can probably work up a patch to defend against this defect, but the underlying problem appears to be AVG anti-virus creating a defective message. The end of the raw message is as follows. The boundary= string and the next two occurrances of that string must be different from the string in the preceding and last boundaries, but it's not.

No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 8.0.238 / Virus Database: 270.11.38/2037 - Release Date: 04/02/09
06:09:00

--=======AVGMAIL-49D503A80000=======
Content-Type: multipart/alternative;
        boundary="=======AVGMAIL-49D503A80000======="

--=======AVGMAIL-49D503A80000=======
Content-Type: text/plain; x-avg=cert; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
Content-Description: "AVG certification"

No virus found in this outgoing message.
Checked by AVG - www.avg.com
Version: 8.0.238 / Virus Database: 270.11.38/2037 - Release Date: 04/02/09 0=
6:09:00

--=======AVGMAIL-49D503A80000=======--

--=======AVGMAIL-49D503A80000=======--

[Mark Sapiro (msapiro)]

The following patch will avoid the problem with these and any similarly defective messages

=== modified file 'Mailman/Handlers/MimeDel.py'
--- Mailman/Handlers/MimeDel.py 2007-10-05 02:52:04 +0000
+++ Mailman/Handlers/MimeDel.py 2009-04-18 20:51:55 +0000
@@ -1,4 +1,4 @@
-# Copyright (C) 2002-2007 by the Free Software Foundation, Inc.
+# Copyright (C) 2002-2009 by the Free Software Foundation, Inc.
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -183,7 +183,7 @@
             try:
                 firstalt = subpart.get_payload(0)
                 newpayload.append(firstalt)
- except IndexError:
+ except (IndexError, TypeError):
                 pass
         else:
             newpayload.append(subpart)

[Mark Sapiro (msapiro)]

For some reason, the patch above is not properly indented. Here it is attached.

[Mark Sapiro (msapiro)]

Only the uncaught TypeError issue related to MM 3.