Mailman's qrunner doesn't provide any way to be monitored

[Tom Haddon]

We currently don't have any way to monitor the health of the qrunner process, besides a check for the process.

Can we add a variable which controls how often something is written to either the xmlrpc or qrunner log file so that if there's no activity within that specified period there will be a "--MARK--" entry with a timestamp. This way an external process (such as nagios) can easily check for the health of the qrunner.

[Tom Haddon (mthaddon)]

This is now made worse by the fact that the only way we can currently monitor mailman is by doing a process listing, and if you have two instances on the same server (such as staging and qastaging) there's no difference in the process listing to be able to tell which one is which.

[Tom Haddon (mthaddon)]

Marking as "high" per latest discussion in Launchpad/IS meeting.

[Barry Warsaw (barry)]

Note that there's actually multiple qrunner processes that must run in order to deliver mail. It would not be hard to add something to the XMLRPC qrunner to write the tag every so often (I think there's even a debug switch that is more chatty to the log). Or you could of course write a small qrunner that just woke up periodically and Did Something. The health of one qrunner does not necessarily imply the health of the entire system though.

[Curtis Hovey (sinzui)]

Hi Tom.

The recent outages were cause when mailman was not syncing with Lp. Mailman reports that it cannot talk to xmlrpc and after a time, it just gives up. I think we want Lp's xmlrpc-malman code to record the timestamp of the last request in the Lp db. I think OSAs know how to pull a timestamp from the db and raise a warning if 30 minutes has passed.

[Francis J. Lacoste (flacoste)]

Hi Curtis,

I think simply logging everytime data has been synced up is more flexible than modifying LP to write to the DB every time a read method is called.

[Curtis Hovey (sinzui)]

Hi Tom and all LOSAs.

Our mailman installation has both an error and xmlrpc log to report actions and exception. I think we want to watch one of these logs using a nagios plugin that I do not understand. The xmlrpc class has a _oneloop() method that manages what happens during a loop and it also catches exceptions and writes them to a log. It could have a rule to log a heartbeat if there were no exceptions during the run.:

    Nov 30 16:55:13 2010 (10236) --MARK--

In the case of the last outage, The --MARK-- would have stopped appear on Dec 01 and something could report mailman xmlrpc was dead. I image we need to decide an acceptable period without a sync. My own expectation is that everything is synced every 10 minutes. 30 minutes becomes a problem for users setting up teams or removing members.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r12199 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/12199>.

[Robert Collins (lifeless)]

mark entries are showing up on qastaging. We need an RT (separate) to get this nagios monitored.

[Robert Collins (lifeless)]

rt 43392