Filtering content while allowing HTML messages impossible

[Psy-q]

I hope this is a genuine bug. I'll use 2.1.7 in these
examples, though I verified this behavior in 2.1.5
also.

I have a mailing list configured as such:

filter_content = 1
filter_mime_types = ''
pass_mime_types = """multipart/mixed
multipart/alternative
text/plain
text/html"""
collapse_alternatives = 0
convert_html_to_plaintext = 0
filter_action = 2

And yet all HTML messages are automatically stripped
of HTML, have their attachments removed and are
immediately posted to the list.

The originals are sent as multipart/alternative with
two attachments, one the text/plain rendition of the
message and one the text/html original. It seems to
be okay, user agents can render this message as
intended. All that remains when delivered by Mailman
is the text/plain bit, though.

Shouldn't this list configuration let HTML mails
through? I can't see a reason why Mailman should
remove the HTML silently with these settings.

If filter_content is turned off, sending HTML
messages works.

(I know it's not polite to send HTML messages through
mailing lists, but some of my users insist.)

[http://sourceforge.net/tracker/index.php?func=detail&aid=1461279&group_id=103&atid=100103]

[Mark Sapiro (msapiro)]

You can't really compare 2.1.5 and 2.1.7 in this respect
since 2.1.5 doesn't have the collapse_alternatives setting
and always operates as if collapse_alternatives is true
(non-zero). I.e., 2.1.5 will always replace a
multipart/alternative part with just the first alternative.

Also, your "immediately posted" remark makes me think that
you think filter_action applies whenever content is filtered
in any way. This is not true. filter_action only applies
when the message is empty after filtering.

You also say "yet all HTML messages are automatically
stripped of HTML, have their attachments removed and ...". I
think you may or may not have a misconception here. With the
pass_mime_types settings you have given, no attachments of
any type other than text/plain or text/html should be left
in the message. The types multipart/alternative and
multipart/mixed only allow such messages/parts to be further
processed looking for allowable types. I.e, with these
settings a message or part of type multipart/related will be
filtered completely (nothing left) regardless of what it
contains, because multipart/related is not accepted.

On the other hand, a message or part of type multipart/mixed
will have it's sub-parts examined for allowable types
because multipart/mixed is allowed.

So the only issue is why are text/html parts being removed
in 2.1.7 from multipart/mixed and/or multipart/alternative
parts. This may or may not be a bug. Please attach full
copies of a message both before and after filtering
including all headers to this report, so we can see if there
is a bug or what the problem might be.

Note that your description of the multipart/alternative
messages and results is the only way Mailman 2.1.5 works so
there is no bug there. In 2.1.7 with collapse_alternatives =
0 and text/html in pass_mime_types, the html should remain
in the outgoing message, so there is clearly a problem here.

[Psy-q (psy-q)]

Thanks for clearing that up, you've confirmed most of what
I've only had vague suspicions about :)

I found out about 2.1.5's hardcoded collapse_alternatives
behavior about half an hour after posting my bug, but
thanks for confirming it.

I feel like a fool at the moment. I just retried this so I
could give you example messages, and the behavior has
stopped. I reproduced the unwanted filtering for over an
hour yesterday, I have no idea why it's working correctly
now. Yesterday, I tried about a dozen combinations of
filter strings and settings, none of which produced the
desired result. In the end, I returned to the
configuration that I posted here and performed a final
test to verify that the behavior is indeed confusing.

Today, I switched my test MTA back on, switched Mailman
back on and tried the very same test message as yesterday.
Amazingly, it went through exactly as expected. Then I
removed text/html from pass_mime_types, and again it
behaved exactly as advertised: It let the
multipart/alternative message through, kept the text/plain
part intact but removed the text/html. This is completely
different behavior than I saw yesterday.

So now I'm even more confused than before -- In theory,
this means that an upgrade to 2.1.7 using this list
configuration would make everything work as it should at
my site, and it would mean that there is no bug.

I will test a bit more thoroughly, I'll remove Mailman and
do a vanilla install of 2.1.7 to see if I can get the
behavior to show again.

[Psy-q (psy-q)]

Seems Konqueror messed up the line breaks in my last reply.
These last two days are full of cursed software :)

I tried a vanilla installation of Mailman 2.1.7, configured
the list as described here and everything worked exactly as
expected (and advertised).

I can't explain where yesterday's behavior came from, and
I'm sorry if I caused any confusion. I can only guess: Could
it be that that something in the 2.1.5 to 2.1.7 upgrade
didn't go right, or that I did something wrong, and somehow
2.1.5's hardcoded collapse_alternatives had taken effect
even though 2.1.7's settings said not to collapse? I did
restart the qrunner with mailmanctl after the upgrade, but
on the other hand, I did install 2.1.7 into the same
location where 2.1.5 previously was. Something there might
have been the source of the error.

[Mark Sapiro (msapiro)]

"Seems Konqueror messed up the line breaks in my last reply."

Actually, it's usually the tracker itself that messes up
line breaks. :-(

"Could it be that that something in the 2.1.5 to 2.1.7
upgrade didn't go right"

When you upgraded from 2.1.5 to 2.1.7 the
collapse_alternatives list settings would have been
initialized from the DEFAULT_COLLAPSE_ALTERNATIVES in
mm_cfg.py/Defaults.py which would have been "Yes" to match
pre 2.1.7 behavior unless you had already set it to No in
mm_cfg.py, but any subsequent display/change on the admin
content filtering page would have shown/set the actual value
at that point.