Hortonworks hdp 2.1 Storm+ZK bundle

Bug #1373006 reported by amir sanjar on 2014-09-23
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juju Charms Collection
Undecided
Unassigned

Bug Description

# Hortonworks Storm & Zookeeper solution
Hortonworks (HDP 2.1.3) Apache Storm is a free and open source distributed
real-time computation system. Storm makes it easy to reliably process unbounded
streams of data, doing for real-time processing what Hadoop did for batch processing
Storm has many use cases: real-time analytics, on-line machine learning, continuous
computation, distributed RPC, ETL, and more. Storm is fast: a benchmark clocked
it at over a million tuples processed per second per node. It is scalable,
fault-tolerant, guarantees your data will be processed, and is easy to set up
and operate.
This charm will build a storm cluster consistent of:
 1. Nimbus master node with following daemons will configured and loaded

> storm-drpc
> storm-logviewer
> storm-nimbus
> storm-ui

 1. Storm worker node(s) with following daemons will configured and loaded:

> storm-logviewer
> storm-supervisor

# Verify deployment
    Zookeeper must be loaded and active, to verify:

    $echo ruok | nc {hdp-zookeeper/0 IP address} 2181
    $echo stat | nc {hdp-zookeeper/0 IP address} 2181

   To verify status of Apache Storm cluster go to:

    http://{nimbus-server ip address}:8080

# Real-time usage
**Example - Deploying and Managing Apache Storm Topologies:**
Following steps demonstrates how to deploy a Storm WordCount application . WordCount application has two parts- Spout randomly generates data
streams and Bolts processes generated stream.

     - $juju ssh nimbus-server/0
     - $storm jar /usr/lib/storm/contrib/storm-starter/storm-starter-0.9.1.2.1.3.0-563-jar-with-dependencies.jar storm.starter.WordCountTopology WordCount

 How to monitor deployment:

     - go to http://{nimbus-server ip address}:8080
     - Under "Topology summary", click on "WordCount"
     - Monitor Spouts & Bolts tasks

# Scale out usage
  Example, adding 5 more worker nodes

     juju add-unit -n 5 storm-worker

   To verify a successful scale:

     - http://{nimbus-server ip address}:8080
     - Under "Topology summary", click on "WordCount"
     - Click on "Spout" link in "Spouts (All time)" section
     - Note "Host" list under "Executors (All time)" section
     - Go back to "Topology summary"
     - Click on "Rebalance" in "Topology actions" section
     - Click on "Spout" link in "Spouts (All time)" section
     - Refresh, notice re-balancing of job as more storm-worker threads become available

# Contact Information
Amir Sanjar <email address hidden>

## Apache & Hortonworks Storm

- [Hortonworks website](http://hortonworks.com)
- [Hortonworks Storm documentation](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_installing_manually_book/content/ch_rpm_storm.html)
- [Apache Storm upstream bug tracker](https://storm.incubator.apache.org)
- [Apache Storm documentation](https://storm.incubator.apache.org/documentation/Home.html)

amir sanjar (asanjar) on 2014-09-23
summary: - Hoetonworks hdp 2.1 Storm+ZK bundle
+ Hortonworks hdp 2.1 Storm+ZK bundle
Jorge Castro (jorge) wrote :

You have some formatting issues in your bundle readme, I've fixed them, see: lp:~jorge/charms/bundles/hdp-storm-zookeeper/readmefixes

Charles Butler (lazypower) wrote :

Greetings Amir,

I've taken some time to review the associated bundle for submission. A few things to note:

There are readme fixes submitted by jorge that should be merged. The bundle itself should be updated to deploy the charms from the charm store resource and not from a personal branch.

This aside, the bundle looks and performs as expected. Once you get the review comments implemented, just let us know and someone will be along shortly to look over the work.

Thanks for the submission! Great work!

Changed in charms:
status: New → In Progress
amir sanjar (asanjar) on 2014-09-26
Changed in charms:
status: In Progress → Fix Committed
Review Queue (review-queue) wrote :

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-1202-results

Charles Butler (lazypower) wrote :

Amir,

This bundle as-is does not pass proof and therefore will not pass - do you think you can implement a quick revision to the bundle that will fix it up and make it G2G? i've tested on AWS and it looks good to me, however it may require additional testing on other substrates with the latest revisions of the charms.

https://gist.github.com/17c3fadbfb69690260ba

Once you get the proof errors passing its ready for a push to the bundle series.

Thanks for the submission and patience during the review process. This will be a welcome addition!

amir sanjar (asanjar) wrote :

done chuck, thanks

Changed in charms:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers