NNTP gatewaying trashes Message-IDs

Bug #266263 reported by Kateturner
18
This bug affects 1 person
Affects Status Importance Assigned to Milestone
GNU Mailman
Status tracked in 3.0
2.1
New
Undecided
Unassigned
3.0
Triaged
Low
Unassigned

Bug Description

when a message is relayed to NNTP, the NewsRunner
unconditionally replaces the existing Message-ID with
its own. this breaking threading when reading a list
via news, and when an NNTP user replies to a list message.

a better solution would be either:
1) do not rewrite the message-id unless the NNTP server
reports an error when posting it (duplicate ID); or
2) optionally, always rewrite posters' message-ids into
a mailman-generated ID, and use this ID for both mail
and news.

i have implemented the former at our site and it
appears to work, but it's so ugly i'd not feel
comfortable submitting a patch for it.

[http://sourceforge.net/tracker/index.php?func=detail&aid=1256272&group_id=103&atid=100103]

Revision history for this message
Nobrowser (nobrowser) wrote :

Finally! I thought I was the only one who noticed this.
See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=346413

Why would the mail message-id ever cause a conflict with
the NNTP server? message-ids are supposed to be globally
unique and there is a convention (in RFC 2822?) that ensures
it (put in the FQDN and something unique to the generating
software). So I think 1) is not ugly but correct :)

2) wouldn't completely solve the problem.
I can post one copy of a message to a mailman list and
another directly to another recipient, including myself.
(I actually Cc myself on all mail I send). If mailman
does 2), the extra recipient will still be confused.

Ian

Revision history for this message
Kateturner (kateturner) wrote :

the conflicts occur when a user cross-posts a message to
two lists. the newsrunner won't turn it into a cross-post,
instead it tries to post it separately to both groups, and
the news server rejects the second one for a duplicate
message-id (tested with innd).

of course, you could say this behaviour is a bug too :) i'm
not sure what the best way to group cross-posted messages
is, since they won't be received by the same mailman
instance. (cancelling the first and re-posting to both
groups might work, but i don't know if inn will still
record the message-id and not allow it to be reused.)

Revision history for this message
poelzi (poelzi) wrote :

Originator: NO

nobrowser: the problem is, that newsserver drop messages in crossposts
when one is a mailinglist and the other is a newsgroup.

i just had an idea. we you use a small sqlite table:

| Original-ID | Translated-ID | Last-Used |

When a message is pulled von news, we replace in the reference header alle
ocurrences of translated-id's with their original id's.
When a message is posted to news, we replace all original-id's with
translated-id's.
A new message still generates a new message id, the original is inserted
into the table along with it's original id.
On all actions, we update last-used.

The daily cron job, deletes all last-used older than x days (60 for
example) to keep the table small. This way, no thread trashing would occure
on both sides, while keeping crossposting possible and overhead isn't too
much i think.

Revision history for this message
phaidros (phaidros) wrote :

We have patched this issue. (Attached)

Newsrunner.py:
Do not replace Message-ID, just add X-Mailman-ID
Skip message if X-Mailman-ID is set
That skips the rewriting of Headers and keeps threading for Mailinglists as well as Newsgroups intact.

a minor fix for stupid MUAs in gate_news:
del 'Newsgroups' header, because MUAs like Evolution take the first useful occurance of a valid Header line to reply. For messages coming from NNTP that is the 'Newsgroups' header. But having that header in a Message received via Mailinglist, leads to a reply towards the newsgroup, even if there a valid To: and/or Reply-To: headers set. Replying to a Newsgroup from a Mailaccount breaks usabilty (user might not even have a nntp account).

hth

Revision history for this message
Ike (icarus) wrote :

Hi!

Best way to handle the crossposting issue would be the following:

Instead of speaking directly with the news server the newsrunner should write the outgoing messages into a queue. This queue is handled by another script that does the queue to news server stuff (eg gate_news?)

When another newsrunner (another mailinglist on the same host) is posting a message it should take a look if a message with the same message-id is already there. (maybe the queue name could be a hash of the message-id). Then it could modify the message with adding a second newsgroup name.

This should handle most problems.

tags: added: mailman3
Revision history for this message
Joshua Cranmer (jcranmer) wrote :

Here is my proposal for what to do:

1. Find all known mailing lists in the to/cc headers, and simultaneously crosspost to those newsgroups
2. Only rewrite a message ID if the news server has one on the server that does not represent this message.

This means that all of the problems with message IDs should be limited to people who put the mailing lists in BCCs, or people who are sending to mailing lists on multiple servers that inject into NNTP in different places.

I do not yet have a patch for either yet.

Revision history for this message
Ike (icarus) wrote :

I solved this problem in another way. Someone wrote for me the SynFU-script: https://github.com/Shirk/SynFU

It is a very good script for synchronizing mails between mailing lists and a newsserver. I'm using it on a system with more than 200 mailing lists and up to 1000 postings a day.

Revision history for this message
Barry Warsaw (barry) wrote :

BTW, thus should be pretty easy now in current mm3 bzr trunk.

tags: added: easy
Revision history for this message
ebullient (ebullient) wrote :
Revision history for this message
Mark Sapiro (msapiro) wrote :

I have marked Bug #496233 as a duplicate of this bug.

Bug #557955 is related to this but is the converse. I.e., instead of not munging message IDs from Usnet, it wants to mung all message IDs.

Revision history for this message
Abhilash Raj (raj-abhilash1) wrote :

This bug has been moved to the new gitlab repo here: https://gitlab.com/mailman/mailman/issues/24

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.