usenet threading improvements

[mdw21]

Basically message-id munging for all messages.

Might be
better as an option? This helps with threading in usenet gating, as
what happens is this:

someone sends a mail with message-
id <email address hidden>
gating replaces the message-id with
<email address hidden>
list member replies via mail,
"in-reply-to" header is set to the <email address hidden>
now when the
message is gated it can't thread because in-reply-to: (which is
translated into references:) is wrong.

Solution is to munge
all message-ids. Doesn't really do any harm AFAICT.

[mdw21 (mdw21)]

The file CookHeaders.patch was added: munging code again...

[elecnix (elecnix-users)]

Logged In: YES
user_id=106549

The submitted patch does not do what it should to. The calls
to time.time() will not return the same thing as in
ToUsenet.py. I don't know much about the internals of
Mailman, and not much either about Python, so I don't really
know where to store the "correct" message ID so that it is
generated once, and used in both places.

[elecnix (elecnix-users)]

Logged In: YES
user_id=106549

I solved the problem by adding the munging code to
DeliverToList, in HandlerAPI.py, right before the message is
sent in the delivery pipeline, and removed it from
ToUsenet.py and CookHeaders.py. This way, the Message-ID
header is calculated only once, and used for all the steps
of the pipeline.

Nicolas Marchildon

[mundaun (mundaun)]

Logged In: YES
user_id=751974

Hello,

we had exactly the same problem. However we run mailman
2.1.2 where things seem to have changed, so I am now writing
down the required steps for more recent releases...

I had to copy the munging code from
Mailman/Queue/NewsRunner.py to
Mailman/Handlers/CookHeaders.py:

block 1:
===
# Matches our Mailman crafted Message-IDs. See
Utils.unique_message_id()
mcre = re.compile(r"""
    <mailman. # match the prefix
    \d+. # serial number
    \d+. # time in seconds since epoch
    \d+. # pid
    (?P<listname>[^@]+) # list's internal_name()
    @ # <email address hidden>
    (?P<hostname>[^>]+) # list's host_name
    > # trailer
    """, re.VERBOSE)
===

block 2:
===
    hackmsgid = 1
    if msgid:
        mo = mcre.search(msgid)
        if mo:
            lname, hname = mo.group('listname', 'hostname')
            if lname == mlist.internal_name() and hname ==
mlist.host_name:
                hackmsgid = 0
    if hackmsgid:
        del msg['message-id']
        msg['Message-ID'] = Utils.unique_message_id(mlist)

===

After that, my biggest problem was that I applied the changes
but mailman didn't follow them. I finally recognised that I had to
restart the qrunner daemon (which wasn't there in older
releases).

Hope this helps
- michael

[Mark Sapiro (msapiro)]

I don't intend to replace the Message-ID: in CookHeaders.py as suggested in this patch, but I will put the original Message-ID in References: and perhaps make other modifications in NewsRunner.py for the posts actually gated to Usenet.

[Mark Sapiro (msapiro)]

The fix adds the original Message-ID: to References: and also adds X-Mailman-Original-Message-ID: and X-Mailman-Original-References: headers for 'forensics'. Hopefully this will help at least partially mitigate threading issues that result from Message-ID: munging.

Also note that Munging the Message-ID in CookHeaders.py as suggested here is not a full solution as there still may be direct To: or Cc: recipients from 'reply all' and these are people likely to generate additional replies so the issue of multiple Message-ID:s is not solved by that approach.