Run log analyzer for all IRC logs

Bug #1203137 reported by Stefano Maffulli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Core Infrastructure
In Progress
Low
Bhuvan Arumugam

Bug Description

It would be good to have an IRC log analyzer examining all the logs in our channels http://eavesdrop.openstack.org/irclogs/. Something like SuperSeriousStats http://sss.dutnie.nl/ for example would run daily and create a stats/ page for each channel.

James E. Blair (corvus)
tags: added: low-hanging-fruit
Revision history for this message
Clark Boylan (cboylan) wrote :

pisg is one alternative that supports supybot logs out of the box (use format 'supy'), but its default config is a bit mean. If this tool is used it will need to be configured to leave out things like 'Big Numbers' and 'Other interesting numbers'.

James E. Blair (corvus)
Changed in openstack-ci:
importance: Undecided → Low
status: New → Triaged
Revision history for this message
shrini (shrini) wrote :

sss seems fine and simple.

Shall I install it and configure on my own VPS server to show stats on irc chats? for a demo?

Or how do you want it to be implemented?

Revision history for this message
James E. Blair (corvus) wrote :

Sure, a demo would be great. If you write a change to the puppet config in the openstack-infra/config repo, we can run it on eavesdrop.openstack.org. See http://ci.openstack.org/ for more information on how to do that.

Revision history for this message
Bhuvan Arumugam (bhuvan) wrote :

sss is simple and support supy format. Here is a demo using #openstack-ceilometer channel.
http://livecipher.com/sss-7.0/%23openstack-ceilometer/

Note: user/history links are broken as my isp don't support SQLite3.

One caveat with sss. It use separate db/config file for each channel.

I'll write a python script to:
  1. curl http://sss.dutnie.nl/sss-7.0.tgz, if it's not installed
  2. install sss (unzip)
  3. foreach channei: # channels may be passed in command line
    if it's a new channel {
       initialize sqlite3 database
      generate conf with right parser, if one is not already generated
      generate conf code vars.php for this channel
    }
  4. generate stats incrementally and make it accessible using this url: http://eavesdrop.openstack.org/stats

New puppet manifest to:
  1. install dependent packages like php, php-pdo (sqlite3), php-mbstring
  2. cron job to run the above python script every hour

Assumption:
  1. sss is setup in the same server that has irclogs

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to config (master)

Fix proposed to branch: master
Review: https://review.openstack.org/72445

Changed in openstack-ci:
assignee: nobody → Bhuvan Arumugam (bhuvan)
status: Triaged → In Progress
Revision history for this message
Bhuvan Arumugam (bhuvan) wrote :

The patch is inline with above notes. new script added to generate the stats using sss. The script is hooked into puppet, to run every hour. The stats are stored in /var/lib/statusbot/www/stats and irc logs for all channels are expected in /var/lib/statusbot/www directory.

Once the patch is applied and script is run, the stats should be accessible using this url:
  http://eavesdrop.openstack.org/irclogs/stats

See help for more details.
ircstats.py -h
usage: ircstats.py [-h] [-V] [-c CHANNEL [CHANNEL ...]] [-v] -l IRCLOG_BASE -t
                   CONF_TEMPLATE -i INSTALL_BASE

optional arguments:
  -h, --help show this help message and exit
  -V, --version version
  -c CHANNEL [CHANNEL ...], --channel CHANNEL [CHANNEL ...]
                        collect stats for these irc channels
  -v, --verbose print debug information
  -l IRCLOG_BASE, --irclog-base IRCLOG_BASE
                        irclog base directory
  -t CONF_TEMPLATE, --conf-template CONF_TEMPLATE
                        template to use for generating conf
  -i INSTALL_BASE, --install-base INSTALL_BASE
                        stats install base directory

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers