no delivery to lists "wrong" footer characters

[Ulrik-haugen]

Mailman 2.1.8:

I recently had a problem with list administrators
changing the footer to something containing characters
that do not fit the default language for the list.
(Actually it was the result of an upgrade but it can be
reproduced more easily this way.)

This resulted in letters to these lists going to the
archive but not beeing distributed to list members.

After some investigation i discovered that they were
instead getting sent to the shunt directory with the
following error message in logs/error:

Jun 15 16:40:26 2006 (11718) Uncaught runner exception:
ASCII decoding error: ordinal not in range(128)
Jun 15 16:40:26 2006 (11718) Traceback (most recent
call last):
  File "/service/mailman/Mailman/Queue/Runner.py", line
111, in _oneloop
    self._onefile(msg, msgdata)
  File "/service/mailman/Mailman/Queue/Runner.py", line
167, in _onefile
    keepqueued = self._dispose(mlist, msg, msgdata)
  File
"/service/mailman/Mailman/Queue/OutgoingRunner.py",
line 73, in _dispose
    self._func(mlist, msg, msgdata)
  File
"/service/mailman/Mailman/Handlers/SMTPDirect.py", line
132, in process
    Decorate.process(mlist, msg, msgdata)
  File "/service/mailman/Mailman/Handlers/Decorate.py",
line 99, in process
    ufooter = unicode(footer, lcset)
UnicodeError: ASCII decoding error: ordinal not in
range(128)

Jun 15 16:40:26 2006 (11718) SHUNTING:
1150382425.6505611+4f10f0ef8067c96952336a07919d018adc6a9979

I think this is a very bad idea as it results in the
list administrator complaining to me as site
administrator about letters not being delivered instead
of (for instance) the list members complaing to the
list administrator about unreadable characters in the
footer.

[http://sourceforge.net/tracker/index.php?func=detail&aid=1507248&group_id=103&atid=100103]

[Mark Sapiro (msapiro)]

I think you are correct. I'm not sure about the fix though.
Current code in Decorate.py is (sorry for wrapped lines):

        # TK: Try to keep the message plain by converting
the header/
        # footer/oldpayload into unicode and encode with
mcset/lcset.
        # Try to decode qp/base64 also.
        uheader = unicode(header, lcset)
        ufooter = unicode(footer, lcset)
        try:
            oldpayload =
unicode(msg.get_payload(decode=True), mcset)
            frontsep = endsep = u''

I see two approaches to fix this. The first is to change

        uheader = unicode(header, lcset)
        ufooter = unicode(footer, lcset)

to

        uheader = unicode(header, lcset, 'ignore')
        ufooter = unicode(footer, lcset, 'ignore')

This will cause the rogue characters to be ignored.

The second approach is to move the

        uheader = unicode(header, lcset)
        ufooter = unicode(footer, lcset)
        try:

inside the try as

        try:
            uheader = unicode(header, lcset)
            ufooter = unicode(footer, lcset)

The result of this will be to add the header and/or footer
in separate MIME parts in this case.

I also thought of using 'replace' instead of 'ignore' in the
first approach, but this will likely result in a
UnicodeError in encode() later on and ultimately have the
same result as the second approach.

[Ulrik-haugen (ulrik-haugen)]

If I understand correctly the solutions you outline are:

1) Have mailman strip the characters outside the list
character set from any header/footer and "attach" the
remainder to the message as before.

2) Attach the header/footer as separate parts if they
contain characters outside the list character set but mark
these parts as beeing encoded with the list character set.

If so, I'd certainly prefer the first approach, stripping
the header/footer over delivering incorrectly marked
messages. One other option might be to skip the
header/footer entirely if it can't be attached correctly.

[Mark Sapiro (msapiro)]

This is now fixed in Subversion both in the trunk and the
Release_2_1-maint branch.

The fix strips (ignores) header/footer characters outside
the character set of the list's language.