Bad Content-Disposition breaks senddigest

[Crotwell]

We are using the default mailman on mac osx server 10.3
which is mailman 2.1.2, so this may be old. But just in
case, we got an email with attachement that caused
mailman to croak in senddigest. Here is the output of
running it by hand. I added 2 prints to see what the
offending filename variable had in it. The
Content-Disposition headers are below. Note that the
two parts of the filename have been joined, but because
of the bizarre endquoting in filename*0, there is an
embedded \n\t.

We fixed this locally by editing the mbox to change the
offending filename to be shorter and not split into two
parts, but it might be nice if mailman would check for
illegal characters embedded in filenames?

In addition to senddigest croaking, all emails after
this bad one were not added to the archive. Luckily we
had made a copy of digest.mbox before running
senddigest manually and were able to manually add the
lost emails to the archive mbox.

roo:/usr/share/mailman root# python -S cron/senddigests
-l chair-archive
(None, None, 'Outside Professional Activities GS
Policy_4-22-03 rev\n\t5-16-05.doc')
(None, None, 'Outside Professional Activities GS
Policy_4-22-03 rev\n\t5-16-05.doc')
Traceback (most recent call last):
  File "cron/senddigests", line 94, in ?
    main()
  File "cron/senddigests", line 86, in main
    mlist.send_digest_now()
  File "/usr/share/mailman/Mailman/Digester.py", line
60, in send_digest_now
    ToDigest.send_digests(self, mboxfp)
  File
"/usr/share/mailman/Mailman/Handlers/ToDigest.py", line
130, in send_digests
    send_i18n_digests(mlist, mboxfp)
  File
"/usr/share/mailman/Mailman/Handlers/ToDigest.py", line
303, in send_i18n_digests
    msg = scrubber(mlist, msg)
  File
"/usr/share/mailman/Mailman/Handlers/Scrubber.py", line
257, in process
    url = save_attachment(mlist, part, dir)
  File
"/usr/share/mailman/Mailman/Handlers/Scrubber.py", line
348, in save_attachment
    fnext = os.path.splitext(msg.get_filename(''))[1]
  File "/usr/share/mailman/pythonlib/email/Message.py",
line 709, in get_filename
    return unicode(newvalue[2], newvalue[0])
TypeError: unicode() argument 2 must be string, not None

--Apple-Mail-5-348793981
Content-Transfer-Encoding: base64
Content-Type: application/msword; x-unix-mode=0666;
        name="Outside Professional Activities GS
Policy_4-22-03 rev
        5-16-05.doc"
Content-Disposition: attachment;
        filename*0="Outside Professional Activities GS
Policy_4-22-03 rev
        5-16-05.do"; filename*1=c

[http://sourceforge.net/tracker/index.php?func=detail&aid=1518281&group_id=103&atid=100103]

[Mark Sapiro (msapiro)]

There are a couple of different issues here.

The '\n\t' in the filename come about because the header is
folded in the middle of the quoted value of the filename*0
parameter. The filename*0 and filename*1 parameters split
the filename per RFC2231, the very purpose of which is to
allow breaking of long parameter values into smaller pieces
so that header folding/unfolding can be accomplished at
syntactic breaks - e.g. at the semicolons.

In this case, either the generating MUA (Apple Mail) folded
the header in the middle of the filename*0 value or it made
the value so long that a subsequent MTA folded it. Thus,
with respect to the '\n\t' in the middle of the name, it
results from a non-compliant message.

The immediate cause of the exception in your case was fixed
in the email library shipped with Mailman 2.1.3. The fix
replaces

            return unicode(newvalue[2], newvalue[0])

with

            return unicode(newvalue[2], newvalue[0] or
'us-ascii')

and makes a similar change in a few other places. The patch
for this change is attached here.

There is also a bug in the (in your case)
/usr/share/mailman/pythonlib/email/Utils.py module. See the
post at
http://mail.python.org/pipermail/email-sig/2006-July/000293.html
for more on this and a possible patch.

[Mark Sapiro (msapiro)]

>We fixed this locally by editing the mbox to change the
>offending filename to be shorter and not split into two
>parts, but it might be nice if mailman would check for
>illegal characters embedded in filenames?

I'm not sure if it is clear from my first comment, but the
cause of this exception (bug) was the RFC2231 splitting, not
the '\n\t' from header folding. Also, in later Mailman
versions, Scrubber is much more robust against bad
characters/encodings in filenames.

>In addition to senddigest croaking, all emails after
>this bad one were not added to the archive. Luckily we
>had made a copy of digest.mbox before running
>senddigest manually and were able to manually add the
>lost emails to the archive mbox.

I suspect these messages were shunted and could have been
reprocessed to the archive by bin/unshunt. At this point,
since the messages are in the archive, you should check to
see if they are still in qfiles/shunt/ and if so, delete
them from there so the don't get inadvertently unshunted in
the future.