Run log analyzer for all IRC logs

[Stefano Maffulli]

It would be good to have an IRC log analyzer examining all the logs in our channels http://eavesdrop.openstack.org/irclogs/. Something like SuperSeriousStats http://sss.dutnie.nl/ for example would run daily and create a stats/ page for each channel.

[Clark Boylan (cboylan)]

pisg is one alternative that supports supybot logs out of the box (use format 'supy'), but its default config is a bit mean. If this tool is used it will need to be configured to leave out things like 'Big Numbers' and 'Other interesting numbers'.

[shrini (shrini)]

sss seems fine and simple.

Shall I install it and configure on my own VPS server to show stats on irc chats? for a demo?

Or how do you want it to be implemented?

[James E. Blair (corvus)]

Sure, a demo would be great. If you write a change to the puppet config in the openstack-infra/config repo, we can run it on eavesdrop.openstack.org. See http://ci.openstack.org/ for more information on how to do that.

[Bhuvan Arumugam (bhuvan)]

sss is simple and support supy format. Here is a demo using #openstack-ceilometer channel.
http://livecipher.com/sss-7.0/%23openstack-ceilometer/

Note: user/history links are broken as my isp don't support SQLite3.

One caveat with sss. It use separate db/config file for each channel.

I'll write a python script to:
  1. curl http://sss.dutnie.nl/sss-7.0.tgz, if it's not installed
  2. install sss (unzip)
  3. foreach channei: # channels may be passed in command line
    if it's a new channel {
       initialize sqlite3 database
      generate conf with right parser, if one is not already generated
      generate conf code vars.php for this channel
    }
  4. generate stats incrementally and make it accessible using this url: http://eavesdrop.openstack.org/stats

New puppet manifest to:
  1. install dependent packages like php, php-pdo (sqlite3), php-mbstring
  2. cron job to run the above python script every hour

Assumption:
  1. sss is setup in the same server that has irclogs

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/72445

[Bhuvan Arumugam (bhuvan)]

The patch is inline with above notes. new script added to generate the stats using sss. The script is hooked into puppet, to run every hour. The stats are stored in /var/lib/statusbot/www/stats and irc logs for all channels are expected in /var/lib/statusbot/www directory.

Once the patch is applied and script is run, the stats should be accessible using this url:
  http://eavesdrop.openstack.org/irclogs/stats

See help for more details.
ircstats.py -h
usage: ircstats.py [-h] [-V] [-c CHANNEL [CHANNEL ...]] [-v] -l IRCLOG_BASE -t
                   CONF_TEMPLATE -i INSTALL_BASE

optional arguments:
  -h, --help show this help message and exit
  -V, --version version
  -c CHANNEL [CHANNEL ...], --channel CHANNEL [CHANNEL ...]
                        collect stats for these irc channels
  -v, --verbose print debug information
  -l IRCLOG_BASE, --irclog-base IRCLOG_BASE
                        irclog base directory
  -t CONF_TEMPLATE, --conf-template CONF_TEMPLATE
                        template to use for generating conf
  -i INSTALL_BASE, --install-base INSTALL_BASE
                        stats install base directory