Hortonworks hdp 2.1 Storm+ZK bundle

[amir sanjar]

# Hortonworks Storm & Zookeeper solution
Hortonworks (HDP 2.1.3) Apache Storm is a free and open source distributed
real-time computation system. Storm makes it easy to reliably process unbounded
streams of data, doing for real-time processing what Hadoop did for batch processing
Storm has many use cases: real-time analytics, on-line machine learning, continuous
computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked
it at over a million tuples processed per second per node. It is scalable,
fault-tolerant, guarantees your data will be processed, and is easy to set up
and operate.
This charm will build a storm cluster consistent of:
 1. Nimbus master node with following daemons will configured and loaded

> storm-drpc
> storm-logviewer
> storm-nimbus
> storm-ui

 1. Storm worker node(s) with following daemons will configured and loaded:

> storm-logviewer
> storm-supervisor

# Verify deployment
    Zookeeper must be loaded and active, to verify:

    $echo ruok | nc {hdp-zookeeper/0 IP address} 2181
    $echo stat | nc {hdp-zookeeper/0 IP address} 2181

   To verify status of Apache Storm cluster go to:

    http://{nimbus-server ip address}:8080

# Real-time usage
**Example - Deploying and Managing Apache Storm Topologies:**
Following steps demonstrates how to deploy a Storm WordCount application . WordCount application has two parts- Spout randomly generates data
streams and Bolts processes generated stream.

     - $juju ssh nimbus-server/0
     - $storm jar /usr/lib/storm/contrib/storm-starter/storm-starter-0.9.1.2.1.3.0-563-jar-with-dependencies.jar storm.starter.WordCountTopology WordCount

 How to monitor deployment:

     - go to http://{nimbus-server ip address}:8080
     - Under "Topology summary", click on "WordCount"
     - Monitor Spouts & Bolts tasks

# Scale out usage
  Example, adding 5 more worker nodes

     juju add-unit -n 5 storm-worker

   To verify a successful scale:

     - http://{nimbus-server ip address}:8080
     - Under "Topology summary", click on "WordCount"
     - Click on "Spout" link in "Spouts (All time)" section
     - Note "Host" list under "Executors (All time)" section
     - Go back to "Topology summary"
     - Click on "Rebalance" in "Topology actions" section
     - Click on "Spout" link in "Spouts (All time)" section
     - Refresh, notice re-balancing of job as more storm-worker threads become available

# Contact Information
Amir Sanjar <email address hidden>

## Apache & Hortonworks Storm

- [Hortonworks website](http://hortonworks.com)
- [Hortonworks Storm documentation](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_installing_manually_book/content/ch_rpm_storm.html)
- [Apache Storm upstream bug tracker](https://storm.incubator.apache.org)
- [Apache Storm documentation](https://storm.incubator.apache.org/documentation/Home.html)

[Jorge Castro (jorge)]

You have some formatting issues in your bundle readme, I've fixed them, see: lp:~jorge/charms/bundles/hdp-storm-zookeeper/readmefixes

[Charles Butler (lazypower)]

Greetings Amir,

I've taken some time to review the associated bundle for submission. A few things to note:

There are readme fixes submitted by jorge that should be merged. The bundle itself should be updated to deploy the charms from the charm store resource and not from a personal branch.

This aside, the bundle looks and performs as expected. Once you get the review comments implemented, just let us know and someone will be along shortly to look over the work.

Thanks for the submission! Great work!

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-1196-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-1202-results

[Charles Butler (lazypower)]

Amir,

This bundle as-is does not pass proof and therefore will not pass - do you think you can implement a quick revision to the bundle that will fix it up and make it G2G? i've tested on AWS and it looks good to me, however it may require additional testing on other substrates with the latest revisions of the charms.

https://gist.github.com/17c3fadbfb69690260ba

Once you get the proof errors passing its ready for a push to the bundle series.

Thanks for the submission and patience during the review process. This will be a welcome addition!

[amir sanjar (asanjar)]

done chuck, thanks