Per-list configurable message headers for digests

[Joseph Fall]

Digest messages are sent out with a partial set of headers for each message.
The list of headers kept for each plain-text digested message is configured by the setting:

PLAIN_DIGEST_KEEP_HEADERS = [
    'Message', 'Date', 'From',
    'Subject', 'To', 'Cc',
    'Message-ID', 'Keywords',
    'Content-Type',
    ]

My users are very perplexed and frustrated by the e-mail headers included in the digest - in particular, the Message-ID and Content-Type are meaningless, and the To header is repetitive. All this "noise" clutters up the content and makes the digest harder to read.

Given that this is a plain-text digest, with the headers listed in the message body, I don't really see an RFC issue here? Yet they can only be configured using a list-wide override in mm_cfg.py

I would be willing to work on this to allow a list admin to override the plain digest headers using the web interface, but want to hear from the development team if this is a patch they would be willing to consider before I invest the time.
thanks.

[Mark Sapiro (msapiro)]

The format of the plain digest is specified in RFC 1153 <http://www.rfc-editor.org/rfc/rfc1153.txt> which says in part "If they exist in the original message header, the following lines must be retained as-is in the reconstructed header: Date:, From:, To:, Cc:, Subject:, Message-ID:, and Keywords:, rearranged to appear in that order."

The only headers included in the plain digest which are not in that list are Message: and Content-Type:. I can understand that Content-Type is not too useful since the message has been scrubbed to text/plain, but it can retain parameters such as format=flowed which might be useful to some.

I would consider removing Content-Type: from the default list, but only after surveying the <email address hidden> list to see if there are objections.

BTW, In current Mailman, the Defaults.py setting for PLAIN_DIGEST_KEEP_HEADERS is

PLAIN_DIGEST_KEEP_HEADERS = [
    'Message',
    # RFC 1153 headers in order
    'Date', 'From', 'To', 'Cc', 'Subject', 'Message-ID', 'Keywords',
    'Content-Type',
    ]

which puts the Subject: in the order specified by the RFC.

[Joseph Fall (powderflask)]

Thanks Mark - excellent info.

It's interesting that the example given in rfc1153 looks exactly like
what I'd like to see - contrary to the actual spec...

Date: ddd, dd mmm yy hh:mm:ss zzz
From: Joe User <username@fqhn>
Subject: Message One Subject

This is the message body of the first message.

Joe

------------------------------

Date: ddd, dd mmm yy hh:mm:ss zzz
From: Jane User <username@fqhn>
Subject: Message Two Subject

This is the body of message two.

Jane
Appreciate the quick reply!

...Joseph
On 12-Nov-11, at 3:41 PM, Mark Sapiro wrote:

> The format of the plain digest is specified in RFC 1153 <http://www.rfc
> -
> editor.org/rfc/rfc1153.txt> which says in part "If they exist in the
> original message header, the following lines must be retained as-is in
> the reconstructed header: Date:, From:, To:, Cc:, Subject:, Message-
> ID:,
> and Keywords:, rearranged to appear in that order."
>
> The only headers included in the plain digest which are not in that
> list
> are Message: and Content-Type:. I can understand that Content-Type is
> not too useful since the message has been scrubbed to text/plain,
> but it
> can retain parameters such as format=flowed which might be useful to
> some.
>
> I would consider removing Content-Type: from the default list, but
> only
> after surveying the <email address hidden> list to see if there are
> objections.
>
> BTW, In current Mailman, the Defaults.py setting for
> PLAIN_DIGEST_KEEP_HEADERS is
>
> PLAIN_DIGEST_KEEP_HEADERS = [
> 'Message',
> # RFC 1153 headers in order
> 'Date', 'From', 'To', 'Cc', 'Subject', 'Message-ID', 'Keywords',
> 'Content-Type',
> ]
>
> which puts the Subject: in the order specified by the RFC.
>
> --
> You received this bug notification because you are subscribed to the
> bug
> report.
> https://bugs.launchpad.net/bugs/889635
>
> Title:
> Per-list configurable message headers for digests
>
> Status in GNU Mailman:
> New
>
> Bug description:
> Digest messages are sent out with a partial set of headers for each
> message.
> The list of headers kept for each plain-text digested message is
> configured by the setting:
>
> PLAIN_DIGEST_KEEP_HEADERS = [
> 'Message', 'Date', 'From',
> 'Subject', 'To', 'Cc',
> 'Message-ID', 'Keywords',
> 'Content-Type',
> ]
>
> My users are very perplexed and frustrated by the e-mail headers
> included in the digest - in particular, the Message-ID and Content-
> Type are meaningless, and the To header is repetitive. All this
> "noise" clutters up the content and makes the digest harder to read.
>
> Given that this is a plain-text digest, with the headers listed in
> the
> message body, I don't really see an RFC issue here? Yet they can
> only be configured using a list-wide override in mm_cfg.py
>
> I would be willing to work on this to allow a list admin to
> override the plain digest headers using the web interface, but want
> to hear from the development team if this is a patch they would be
> willing to consider before I invest the time.
> thanks.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/mailman/+bug/889635/+subscript...

[Joseph Fall (powderflask)]

More information about RFC 1153...

This is an "experimental" RFC, and a fairly old one at that.

The IETF definition:

"Experimental RFCs are for specifications that may be interesting, but for which it is unclear if there will be much interest in implementing them, or whether they will work once deployed. That is, a specification might solve a problem, but if it is not clear that many people think that the problem is important, or think that they will bother fixing the problem with the specification, the specification might be labeled an Experimental RFC. If, later, the specification becomes popular (or proves that it works well), it can be re-issued as a standards-track RFC. Experimental RFCs are also used to get people to experiment with a technology that looks like it might be standards-track material, but for which there are still unanswered questions. "

http://www.ietf.org/tao.html#anchor45

Given its vintage and the decline of e-mail digests, seems unlikely to ever make it to "standards-track. So, while it does make a very reasonable choice for a default, I don't see there's much of a case for forcing list owners to adhere, especially for plain-text digests.

So I'll pose the question again - if I push an enhancement to allow the DIGEST_KEEP_HEADERS to be administered on a per-list basis, would you be willing to review and consider merging it in?

thanks Mark.

[Joseph Fall (powderflask)]

Browsing the source for the 3.x series, it looks like one possible way to handle this might be to add a new digest format - say "Flexible Plain Text Digest".

class FlexibleTextDigester(RFC1153Digester):

would simply override a few things in the init() method to pull in user-defined alternatives to the RFC1153 defaults from the mlist object to override the config settings, but otherwise rely on the base digester to do the work.

That would ensure no existing lists using RFC1153 plain text digests would be affected in any way.
Of course MailingList(Model) would need to define a new field(s) - DB schema change - that's a bummer.

I saw a few other feature requests for non-compliant RFC1153 digests that might be consolidated with this approach?

BTW - looks like there is a minor coding error (v. 3.0.0a8+ ) on line 242 in runners.digest.py
class RFC1153Digester(Digester):
      def add_message(self, msg, count):
         ....
         for header in config.digests.plain_digest_keep_headers.split():
             ...

should read

         for header in self._keepers:

I doubt this causes a bug at present, but has the potential to, if I've understood the code correctly.

[Abhilash Raj (raj-abhilash1)]

This bug has been moved to the new gitlab repo here: https://gitlab.com/mailman/mailman/issues/33