Mail <-> News gateway misses some messages

[Dâniel Fraga]

I have several lists gatewaying to nntp groups and I notice that not every message posted to the nntp groups appear on the respective list. For example, if you access:

Server: news.abusar.org
Group: u-br.comp.so.linux
Subject: Flash Player 10 - prerelease alpha version 64 bit for Linux!
Message-ID: <email address hidden>

There're a lot of replies to this thread, but on he mailman archive we see just 4 replies:

http://teleporto.abusar.org/pipermail/u-br.comp.so.linux/2008-November/thread.html

Notice that the archive is from the begining of november until Nov 17, 19:05 (at least the time I saw)

What could be happening? I have this in mailman cron:

# At 8AM every day, mail reminders to admins as to pending requests.
# They are less likely to ignore these reminders if they're mailed
# early in the morning, but of course, this is local time... ;)
0 8 * * * /usr/bin/python -S /usr/local/mailman/cron/checkdbs
#
# At 9AM, send notifications to disabled members that are due to be
# reminded to re-enable their accounts.
0 9 * * * /usr/bin/python -S /usr/local/mailman/cron/disabled
#
# Noon, mail digests for lists that do periodic as well as threshhold delivery.
0 12 * * * /usr/bin/python -S /usr/local/mailman/cron/senddigests
#
# 5 AM on the first of each month, mail out password reminders.
0 5 1 * * /usr/bin/python -S /usr/local/mailman/cron/mailpasswds
#
# Every 5 mins, try to gate news to mail. You can comment this one out
# if you don't want to allow gating, or don't have any going on right now,
# or want to exclusively use a callback strategy instead of polling.
*/42 * * * * /usr/bin/python -S /usr/local/mailman/cron/gate_news

#
# At 3:27am every night, regenerate the gzip'd archive file. Only
# turn this on if the internal archiver is used and
# GZIP_ARCHIVE_TXT_FILES is false in mm_cfg.py
27 3 * * * /usr/bin/python -S /usr/local/mailman/cron/nightly_gzip

And I have those processes running:

 2568 ? Ss 0:00 /usr/local/bin/python /usr/local/mailman/bin/mailmanctl -s start
 2581 ? S 60:31 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=ArchRunner:0:1 -s
 2582 ? S 0:24 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=BounceRunner:0:1 -s
 2583 ? S 0:18 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=CommandRunner:0:1 -s
 2584 ? S 1:46 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=IncomingRunner:0:1 -s
 2585 ? S 0:22 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=NewsRunner:0:1 -s
 2586 ? S 1:03 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=OutgoingRunner:0:1 -s
 2587 ? S 0:24 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=VirginRunner:0:1 -s
 2588 ? S 0:00 /usr/local/bin/python /usr/local/mailman/bin/qrunner --runner=RetryRunner:0:1 -s
27405 ? SN 0:00 /usr/bin/python -S /usr/local/mailman/cron/gate_news

It's strange because the ArchRunner line seems to be stuck... or not? 60:31 is normal?

Thanks.

[Mark Sapiro (msapiro)]

It is normal for ArchRunner to use way more CPU than the other runners.Whether or not 60 minutes is "normal" depends on how long Mailman has been up and how many messages it has processed.

As far as your missing messages from usenet are concerned, what does Mailman's fromusenet log say?

Note that if a message retrieved from the newsgroup contains an X-BeenThere: header with the name of the list, it won't be posted back to the list. When messages are gated from the list to the newsgroup, that header is added. If someone then posts a reply to that message on the newsgroup using a news agent that doesn't remove that header, the reply won't come back to the list. Possibly, one or more people is using such a broken agent. Note that I am not specifically aware of such agents. I'm just guessing.

[Dâniel Fraga (fragabr)]

Hmm interesting what you said. I'll check. Anyway, the fromusenet log has just the normal stuff:

Nov 17 19:42:02 2008 (18390) u-br.comp.so.linux: [2..97123]
Nov 17 19:42:02 2008 (18390) gating u-br.comp.so.linux articles [97123..97123]
Nov 17 19:42:02 2008 (18390) posted to list u-br.comp.so.linux: 97123
Nov 17 19:42:02 2008 (18390) u-br.comp.so.linux watermark: 97123

I'll pay attention to X-BeenThere header.

[Dâniel Fraga (fragabr)]

I checked a thread with missed messages. The following thread was posted by me directly using a nntp reader, The replies also were posted by nntp readers. Nobody posted through mailman (so there's no X-BeenThere headers on the thread):

http://teleporto.abusar.org/pipermail/u-br.comp.so.linux/2008-November/thread.html

You'll notice that there're just a few messages on the mailman archive. There are my messages that are missing too (I use Claws-mail and it doesn't seem to be broken). The fromusenet log doesn't show anything wrong.

Server: news.abusar.org
Message-ID: <email address hidden>

It's really a very strange issue.

[Mark Sapiro (msapiro)]

You say "The fromusenet log doesn't show anything wrong.".

Are you saying that there are log messages like "posted to list u-br.comp.so.linux: 97123" for every message that should be posted to the list, including the ones missing from the list, or are the missing messages also missing their "posted to list" fromusenet log entries?

[Dâniel Fraga (fragabr)]

First option. There are log messages like "posted to list u-br.comp.so.linux: 97123" for every message that should be posted to the list, including the ones missing from the list.

[Mark Sapiro (msapiro)]

I think it is unlikely that anything is going wrong in gate_news, but just in case, make sure that Mailman's crontab has a valid

MAILTO=

setting or if not, make sure that mail to mailman@... is actually deliverable to someone and is not for example being silently discarded by the mailman site list.

The "posted to list u-br.comp.so.linux:" fromusenet log entry is written by gate_news immediately AFTER it has successfully queued the message in Mailman's in/ queue with listname='u-br.comp.so.linux' and fromusenet=1 in the queue entry's metadata. The fromusenet=1 flag will bypass list membership and moderation and "implicit destination" checks, but all other message holds still apply. Content filtering is also applied.

Have you checked to see if the messages are waiting moderator action? Have you checked Mailman's vette log to see if messages are being held.

It is also possible that content filtering is removing the entire message which can result in the message's being silently discarded depending on the setting of filter_action.