New charm: apache-flume-twitter

[Kevin W Monroe]

This charm provides a Flume agent designed to process tweets from the Twitter Streaming API and send them to the apache-flume-hdfs agent for storage in the shared filesystem (HDFS) of a connected Hadoop cluster. This leverages the TwitterSource jar packaged with Flume.

Big Data charmers need to review the current state of this charm and verify its readiness for the charm store.

Charm store link: https://jujucharms.com/u/bigdata-dev/apache-flume-twitter/trusty

[Cory Johns (johnsca)]

Twitter link in README should point to https://apps.twitter.com/ because it was not clear how to get from https://dev.twitter.com/user/login to there to create the tokens.

Twitter source generates Avro events, which are then wrapped in another layer of Avro when being sent to HDFS. Ideally, we could find a way to send them directly w/o re-wrapping them.

[Kevin W Monroe (kwmonroe)]

Thanks Cory, I updated the link to apps.twitter.com.

I've also switched the default flume-hdfs serializer to 'text' instead of 'avro'. The tweets are still in avro format (because that's how the TwitterSource jar does it), but we are no longer re-serializing them once they hit flume-hdfs.

[Cory Johns (johnsca)]

I added some additional status reporting when the API credentials are not provided, but with that minor addition, this looks good. +1

[Kevin W Monroe (kwmonroe)]

Thanks Cory! I've frozen the source in the related ~bigdata-charmers staging branch. Assigning this to 'nobody' to get a ~charmer's blessing.