Daily Status Report script...

[shub]

Folks,

I quickly whacked together a Daily Status Report script for
Mailman (using Bourne shell, not Python ;), and thought that other
folks might be interested in seeing it.

The basic concept is a program that gets fired off at 23:59 every
night, and goes through a variety of log files looking for entries
specific to that date, and indicating problems or certain types of
activity that might be of interest to someone trying to administer
the server. It also does an "ls -la" of /usr/local/mailman/qfiles/*,
so that you can see what is in the queue at the time of the running
of the script.

My concept was that this daily report would get e-mailed to the
admin, or posted to a "reports" mailing list, where they could be
archived and kept for future reference.

The script does not (yet) do any statistics calculations, although it
should be relatively easy to hack together some basic stats using
awk, sort, etc....

Anyway, I thought I'd share it and let folks take a look at it, and if
anyone has any recommended improvements, we can incorporate
those and share them back out with everyone.

The code is written under a BSD-style license, so if you don't want
to contribute any changes back to me, that's okay. Of course, I
would prefer that you did, but I leave the choice up to you.

[tgc99 (tgc99)]

Logged In: YES
user_id=1159458

The current UID grab command doesn't work on Solaris (2.6 &
8 tested).

I'd recommend this instead:
ps -o user -p $$|tail -1

This is tested and works on RH 6.2, RH 7.3, RHEL 2.1, RHEL3,
FC3, FreeBSD 4.9, Solaris 2.6, 8.

[adrianwi (adrianwi-users)]

Logged In: YES
user_id=1175103

Use of variable named UID does work well with OS X (version
10.2.8). Apparently the variable UID is a constant already
in use. When trying to the run the script without
modification, I was receiving the following error message:

UID: readonly variable

This issue was resolved by changing the name of variable,
UID, to something else, such as MMUID. Works fine with this
change.

As an aside (& for what it is worth), the UID grab command
suggested by tgc99 on 2005-02-16 03:15 works on this system
(OS X - version 10.2.8)

[shub (shub-users-sf)]

Logged In: YES
user_id=18417

The UID variable in the current code was already replaced by MYUID,
because I got complaints on other platforms. But UID wasn't available to
me as a useful constant, so I had to use something else to obtain the
value.

The recommended patch from tgc99 does work, and I will be uploading a
new version of the code soon.

[shub (shub-users-sf)]

Logged In: YES
user_id=18417

Okay, I've deleted the old version of the mmdsr script that was attached,
and uploaded the latest version. This is what we're currently using to
monitor the lists on python.org, and we have found it very useful.

Any comments you may have will be appreciated.

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998

I tried the script and other than the expected site specific
changes, I found two issues which are both addressed by the
patch in the mmdsr.patch file.

1) Mailman's log files are not necessarily in
$var_prefix/logs/ - they can be in any directory. The patch
adds a LOGDIR directory independant of MMDIR.

2) The $TMPLOG file is not removed at completion. The patch
removes it.

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998

Oooops! I mistakenly thought I could add the mmdsr.patch as
a downloadable file, but I can't so here it is. Watch out
for wrapped lines ...

--- mmdsr.orig 2005-09-06 20:37:53.000000000 -0700
+++ mmdsr 2005-09-10 18:33:14.532393572 -0700
@@ -132,6 +132,7 @@

 MMDIR="/usr/local/mailman"
 TMPDIR="/tmp"
+LOGDIR="/var/log/mailman"

 ###############################################################################
 # Maximum number of subdirectory entries to display in report
@@ -234,9 +235,9 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
- $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG
+ $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG

- if [ -f "logs/${LOG}" ] ; then
+ if [ -f "$LOGDIR/${LOG}" ] ; then

  if [ "${LOG}" = "post" ] ; then

@@ -304,9 +305,9 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
- $GREP -si "^$DAY [0-9][0-9:]* $YEAR" logs/$LOG >> $TMPLOG
+ $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG

- if [ -f "logs/${LOG}" ] ; then
+ if [ -f "$LOGDIR/${LOG}" ] ; then

  if [ "${LOG}" = "error" ] ; then

@@ -453,4 +454,4 @@
     $CAT $TMP
 fi

-$RM $TMP
+$RM -f $TMP $TMPLOG

[shub (shub-users-sf)]

Logged In: YES
user_id=18417

Okay, I took Mark's comments and incorporated them. Adrian Wells also
pointed out a log file difference between Mailman 2.1.5 and 2.1.6 that
caused the summary of the "smtp" log to be munged.

I've now fixed these bugs, deleted the old file, and uploaded the new one
(version 0.0.12).

Thanks!

[tgc99 (tgc99)]

Logged In: YES
user_id=1159458

ps output on solaris is full of whitespace but a further
echo get's rid of it.
The lines in the smtp log are sometimes broken up by a
newline (right before the msgid) which throws of the
summary. Piping it through sed first will rejoin the broken
lines.
Use $AWK instead of awk.

Patch inserted below:
--- mmdsr.orig 2005-10-19 09:42:30.000000000 +0200
+++ mmdsr 2005-10-19 09:44:23.000000000 +0200
@@ -203,7 +203,8 @@
 # there is an easier cross-platform way to do it,
please let me know.
 ###############################################################################

-MYUID=`$PS -o user -p $$ | $TAIL -1`
+GRABUID=`$PS -o user -p $$ | $TAIL -1`
+MYUID=`echo $GRABUID`
 RUNAS="mailman"

 ###############################################################################
@@ -254,7 +255,7 @@
     $TOUCH $TMPLOG
     echo "Log file: $LOG" >> $TMP
     echo "==============================" >> $TMP
- $GREP -si "^$DAY [0-9][0-9:]* $YEAR" $LOGDIR/$LOG >>
$TMPLOG
+ $SED -e :a -e '$!N;s/\n //;ta' -e 'P;D' $LOGDIR/$LOG |
$GREP -si "^$DAY [0-9][0-9:]* $YEAR" >> $TMPLOG

     if [ -f "$LOGDIR/${LOG}" ] ; then

@@ -264,7 +265,7 @@
            echo "Hourly Summary of Posts" >> $TMP
            echo "-----------------------" >> $TMP

- $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | awk '{ printf(
"%8d %02d:00-%02d
:59\n", $1, $2, $2 ) }' >> $TMP
+ $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.*$//' $TMPLOG | $UNIQ -c | $SORT -n +1 | $AWK '{
printf( "%8d %02d:00-%02
d:59\n", $1, $2, $2 ) }' >> $TMP

            echo "" >> $TMP
            echo "Post Count by List" >> $TMP
@@ -295,7 +296,7 @@
            echo "" >> $TMP
            echo "Hourly Summary of Messages Sent" >> $TMP
            echo "-------------------------------" >> $TMP
- $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | awk '{
val=int($1); sum[val]+=$2
 } END { for (i=0; i<24; i++) { printf "%8d
%02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP
+ $SED -e 's/^[A-Z][a-z][a-z] *[0-9]* //' -e
's/:.* for / /' -e 's/ recips,.*$//' $TMPLOG | $AWK '{
val=int($1); sum[val]+=$
2 } END { for (i=0; i<24; i++) { printf "%8d
%02d:00-%02d:59\n", sum[i], i, i } }' >> $TMP

        else

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998

The vette log summary lists posts held for moderation
individually under "Other Errors:". The following patch
(watch out for wrapped lines) summarizes them by list instead.

--- mmdsr 2005-09-29 21:31:33.000000000 -0700
+++ mmdsrx 2005-11-06 08:20:58.835895418 -0800
@@ -415,10 +415,15 @@
      echo "------------------------------" >> $TMP
      $GREP -i 'Posting to a moderated newsgroup' $TMPLOG |
$AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP

+ echo "" >> $TMP
+ echo "Post to moderated list (by list):" >> $TMP
+ echo "------------------------------" >> $TMP
+ $GREP -i 'Post to moderated list' $TMPLOG |
$AWK '{ print $6 }' | $SORT | $UNIQ -c | $SORT -nr >> $TMP
+
      echo "" >> $TMP
      echo "Other Errors:" >> $TMP
      echo "------------------------------" >> $TMP
- $EGREP -vi '(Post by non-member|suspicious
header|message approved|Discarded posting|bulk message
discarded|junk message discarded|Message has implicit
destination|Posting to a moderated newsgroup|Message
discarded, msgid)' $TMPLOG | $SED 's/^.* ([0-9]*) //' |
$SORT | $UNIQ -c | $SORT -nr >> $TMP
+ $EGREP -vi '(Post by non-member|suspicious
header|message approved|Discarded posting|bulk message
discarded|junk message discarded|Message has implicit
destination|Posting to a moderated newsgroup|Post to
moderated list|Message discarded, msgid)' $TMPLOG | $SED
's/^.* ([0-9]*) //' | $SORT | $UNIQ -c | $SORT -nr >> $TMP

  else

[shub (shub-users-sf)]

The file mmdsr was added: mmdsr -- version 0.0.15

[shub (shub-users-sf)]

The file mmdsr.readme was added: mmdsr readme

[shub (shub-users-sf)]

Logged In: YES
user_id=18417

Folks,

Okay, I just uploaded version 0.0.15, which addresses the most recent
comments here from Mark and Tom, and makes a few additional tweaks as
well. See the version comments at the top of the script.

[daveyost (daveyost-users)]

Logged In: YES
user_id=330919
Originator: NO

I suggest incorporating the readme into the shell script, so there is only the one file.

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998
Originator: NO

File Added: mmdsr

[msapiro (msapiro-users)]

The file mmdsr was added: mmdsr -- version 0.0.16

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998
Originator: NO

I have added two files.

Version 0.0.16 corrects a couple of typo/minor issues. It is the version that has been in the contrib directory of the distribution since Mailman 2.1.8.

Version 0.0.17 processes two generations of rotated logs (assuming they are named xxx and xxx.1 and xxx.1 exists) so that data is not lost on the day the logs rotate. It also changes a deprecated sort key option to one that should work with all sort commands. This version will be in the contrib directory of the Mailman 2.1.10 distribution.

File Added: mmdsr

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998
Originator: NO

Re-did the 0.0.17 change to use the prior sort key field separator by default but to make it a user settable parameter.
File Added: mmdsr

[msapiro (msapiro-users)]

The file mmdsr was added: mmdsr -- version 0.0.17

[msapiro (msapiro-users)]

Logged In: YES
user_id=1123998
Originator: NO

The following changes have been made in version 0.18

# 0.0.18 Update by Mark Sapiro <email address hidden>
# Updated on: Tue May 13 08:58:26 PDT 2008
# Changes for Mailman 2.1.10 --
# Added 'bad' to the list of queues.
# Made listing of a queue directory conditional on existence.
# Sumarized unparseable messages and preserved queue entries
# in the Other Errors section.

File Added: mmdsr

[msapiro (msapiro-users)]

The file mmdsr was added: mmdsr -- version 0.0.18

[msapiro (msapiro-users)]

The file mmdsr was added: mmdsr -- version 0.0.19

[msapiro (msapiro-users)]

Posted version 0.0.19 which corrects a minor error in the 'posts from usenet' report.

[tlitt (tlitt)]

It would be nice if this only generated output when it found a problem. Less likely to be ignored. (Actually, what I often do in such scripts is mail only on error, except even if no error 1x/week -- so I know that the monitor is alive.)

Failing that, for low activity sites, if it could run less frequently than daily (say, weekly) without getting confused.