Charm needed: logstash

[Sidnei da Silva]

With Orchestra supporting centralized logging, I thought it would be neat if we also had centralized log searching by feeding data into logstash and plugging it back into ElasticSearch (Bug #822979).

Here's a document with a minimal scalable cluster: http://logstash.net/docs/1.1.0/tutorials/getting-started-centralized

Instead of colocating the logstash log shipper, it could receive data from the syslog server via tcp or udp, perhaps with haproxy in front of it to load balance between multiple log shippers.

The architecture would look like:

[N logging servers] -<round-robin>- [N haproxy] -<load-balance>- [N logstash shippers] -<(multiple?) amqp>- [N logstash indexers] -> [1 elasticsearch cluster]

Where N is not necessarily the same at each level, but you start with 1 and go adding nodes to scale it out. The elasticsearch cluster can and should have more nodes as well, but I'll leave hashing that out to the elasticsearch charms.

[Paul Czarkowski (paulcz)]

I've put together a basic charm for logstash ... using the standalone install for now, but the idea will be to build additional charms for connecting it to elastic search and a queuing system ( amqp or redis ) and kibana. it's waiting approval, chances are I'll need to do some more cleanup when the charmers group look over it and point out all my bad practices :)

https://bugs.launchpad.net/charms/+bug/1073801

[Andrew Farrell (ahfarrell)]

I can help with this charm if you would like

[José Antonio Rey (jose)]

Hey, Andrew!

I see from a previous comment that a logstash-indexer has already been done, and a logstash-agent charm is also on the store. If there's something missing you're totally welcome to go ahead and do it! :)