statusbot needs better logging

[Anita Kuno]

Statusbot goes down and the current logging is inadequate to acertain why. This situation could stand some improvement.

[Anita Kuno (anteaya)]

<fungi> seems we've lost gerritbot and statusbot. giving them both a swift kick

<fungi> so looks like we lost openstackgerrit to a ping timeout today at 09:38:36 (but the process is still running on the server), and openstackstatus disapeared at 10:20:51 yesterday closing its connection (process is not running at all on the server)

<anteaya> utc time?
<fungi> yeah. i'm checking their logs before i (re)start them
<anteaya> k thanks
<anteaya> are their logs accessible publicly
<anteaya> would love to look

<fungi> no, they're not. but also nothing of interest in the gerritbot logs. dug around a good 10 minutes before the timeout all the way up through when it fell off the server and no indication in the logs that it ever knew anything out of the ordinary happened
<fungi> still maintained its connection for the gerrit event stream and was receiving events and thought it was announcing them

<fungi> and statusbot's logs (even the debug log) just stop at the point where the process died

<fungi> no error, no traceback, nothing

* openstackstatus (~<email address hidden>) has joined #openstack-infra

<anteaya> funny nothing showed up in the logs
<ttx> openstackstatus seems like the bot which should always be up and it's down half the time :)
<fungi> ttx: yes. perhaps we need an openstackstatusstatus bot to announce the status of openstackstatus
<anteaya> ha ha ha
<anteaya> does openstackstatus need better logging so that when it is down, we have a clearer picture as to why?

<fungi> or someone to leave it running under a debugger sometime and catch it when it dies. i suspect it wouldn't drag down performance much given its activity leve
<fungi> l
<fungi> anteaya: yes