Charm needed: MapR hadoop distribution

[David Tucker]

Open project for new Charm to support MapR technologies Hadoop Distribution.

[David Tucker (dbtucker)]

Uploaded base version to lp:~dbtucker/maprcharm/trunk

[Quinton Anderson (quintona)]

I would like to get this working so that I can show a model being built at scale (RF) and then scored at scale using Storm.

[Mark Mims (mark-mims)]

Hi Quinton,

Reviewing now. Great idea, we should get the mapr charm related to the storm charm... I guess the best way to do that is to add a relation that's essentially a mapr-spout for storm... not sure. Let's get the basic charms in there first. Please take a look at the existing storm charm to plan out what that realation should look like. It's not in the store yet, but it's in the works at lp:~maarten-ectors/charms/precise/storm/trunk

Thanks!

[Mark Mims (mark-mims)]

Hey David,

Thanks for the contribution!

Couple of minor things to clean up first... then we can get on to some suggestions on structure.

- this is small, but we need to get an icon.svg in the root directory of the charm for the gui and web-based store. Easiest way to do that is to `charm create foo` and copy the generated icon.svg file from there. Edit this in inkscape or cs.

- there are lots of relations defined in the metadata that don't have active hooks. There's a difference between a relation purposefully using default hooks and a relation that's just still TODO. If they're TODO, please comment them out in the metadata so the only active relations are actually implemented. This prevents people trying to wire up stuff that doesn't exist yet.

- the README needs to provide a working example of using the charm. Charm naming in the README isn't the same as the name in metadata.yaml and config options need to be passed as yaml, not direct roles files. The README should really provide more info about various config options too... even if it's just links to real mapr docs.

Ok, now as for roles and relations. Let's figure out how to implement the roles as juju relations. Even if the best way to install everything is the current installer and its roles files... let's have juju define those roles (and writing the roles files) based on relations that are made. This one's a big change and is worth dedicating some time for hangouts to get this done and in the store.

Thanks!
Mark

[Mark Mims (mark-mims)]

bug in the queue requires this to be incomplete instead of in progress. We'll pick this up out-of-band from the bug report, but "Fix Committed" sticks it back on the queue for more attention.

[David Tucker (dbtucker)]

Simplified charm model ... only supporting MapR M3 configurations for now.

Removed unsupported "provides" hooks. The cluster is self-contained via peer relations ... no need for master/slave relations to support MapR cluster.

[Antonio Rosales (arosales)]

Per comment 6 moving bug to Fix Commited for review in the Charm Queue.

[Marco Ceppi (marcoceppi)]

Hi David! Thanks for you patience during this review process. I've begun another review on this now and will report back with my findings

[Marco Ceppi (marcoceppi)]

Hello David!

# Proof

I've run `juju charm proof` from charm-tools with the following output:

W: all charms should provide at least one thing

Though, from previous reviews it appears we'll be working on figuring out how best mapr charm fits in to the charm ecosystem.

# Review

Your scripts are calling hooks/lib/juju.sh but this file doesn't exist. I'd also recommend adding `set -eux` to the top of your hooks, this will allow better debug information.

# Knitpicks

I'd recommend naming the charm "mapr" instead of "maprcharm" since the charm portion is implied

# Comments

I honestly can't find much else to poke at. However, given how close mapr is to hadoop, etc it might be interesting to provide the a "mapred" interface (http://manage.jujucharms.com/interfaces/mapred) much like the hadoop charm, so it can be plugged in to hive and other charms that consume that interface. I apologize that mapr and hadoop aren't my strongest points of understanding so I may be off base with this suggestion. If you could clarify the points above then we can move forward with getting this in the store!

Thanks for your time and patience with this submission we appreciate all the efforts you've put forward in getting mapr made available through juju!

[Marco Ceppi (marcoceppi)]

I've moved this bug to Incomplete pending the above questions being answered. When you're ready move the bug's status to either "new" or "fix committed" to have it placed back in the queue.

[Antonio Rosales (arosales)]

Quick update, discussing with David Tucker on next steps needed.
-Antonio

[David Tucker (dbtucker)]

I think all the files are now properly checked in and validated.

For the short term, I have a default NFS service provided by the Charm (since the MapR distribution supports full read-write NFS). The connection to the mapred service is a little more complicated, because I have to coordinate the installation of the mapr-client package and the configuration. I'll work on that for the next major release.

[Antonio Rosales (arosales)]

David,

Thanks for your continued work on this charm. Look at lp:~dbtucker/maprcharm/trunk it looks like that is where you have been committing to. Please confirm if this is the appropirate branch.

Once you are ready for the this branch to be reviewed just set the "Status" of the bug from "Incomplete" to "Fix Commited" and it will show up in the charm review queue @ http://manage.jujucharms.com/tools/review-queue and we'll get your a promt review.

-thanks,
Antonio

[Charles Butler (lazypower)]

Greetings David,

Thank you for your continued work on this charm. Its progressing nicely, and the simplicity in approach to configuring the service leveraging MapR's existing list files is a nice touch. I've been reviewing the charm as is and I have some notes below:

Following along from the readme, the example instructions need some tweaks to work with recent revisions of Juju. I've tested this on 0.16.5 and 0.17.1 - using -constraints="instance-type=m1.xlarge" maprcharm MyCluster - 2 things with this statement. The new format of constraints if cpu=X mem=Y which I see you have illustrated in the test, but not the docs. And MyCluster seems to not be a valid service name, changing to "mycluster" worked a treat.

After the service was deployed and scaled, the web service was not configured correctly. Visiting http://ip:8443 the web server sent the server side include as a download. I restarted all the daemons on the system and still was unable to view the web management interface.

I'm going to move the status of the charm to "incomplete", and when these items have been cleared up set the status to "new" or "needs review" and someone will be along shortly to re-review the charm.

Thanks

[Antonio Rosales (arosales)]

David,

Moving this bug to "fix commited" so it may be reviewed the by the ~charmers team. Please confirm lp:maprcharm has the branch you would like to be reviewed.

-thanks,
Antonio

[Antonio Rosales (arosales)]

Confirmed with David Tucker that lp:maprcharm is up to date and ready for review based off the last review.

-thanks,
Antonio

[Charles Butler (lazypower)]

David,

Thank you for the continued effort.

I'm still seeing the same behavior when following the README documentation. Port 8443 of the master node continues to send a response with an incorrect MIME type, downloading a file from the host instead of displaying the control panel.

Port 7221 however shows me a Container Location Database status display.
http://i.imgur.com/kRT0Lsr.png

Port 50030 displays a job tracker display
http://i.imgur.com/ZtAGCMp.png

Port 50060 displays a machine read out on the hadoop node task tracker
http://i.imgur.com/8pMh67y.png

Judging on the README instructions I am unclear of what the expected result is to be if not a web based interface. Can you validate these findings?

In addition you provide a default password for the application in config.yaml. This is against charm store policy as it opens an attack vector on that service if not reconfigured. Traditionally authors leave the password field blank and fail to configure the service if no password is supplied. An alternative solution would be to generate the password randomly on deploy, and expose a password field that the user can then update - allowing the service to reconfigure the login based on the users input.

You can review the charm store policy here: https://juju.ubuntu.com/docs/authors-charm-policy.html

barring these modifications the charm is extremely close to making it into the charm store. Thank you for your continued effort with the MapR charm. I look forward to the next round of revisions.

I'm going to move the status of the charm to "incomplete", and when these items have been cleared up set the status to "Fix Committed" or "needs review" and someone will be along shortly to re-review the charm.

[David Tucker (dbtucker)]

Charles,

The primary access to the cluster is over https, not http. The MapR software deliberately does not redirect http:// requests, you you must use https://<node-1>:8443 . I've updated to the documentation to emphasize this fact a bit more.

Regarding the password: is there no way to check this before the "deploy" step ? It seems a bit rude to wait till <n> instances are spun up and running before notifying the user that "by the way, you need to specify a password on the command line ". The Charm documentation even suggests requiring no specific configuration (https://juju.ubuntu.com/docs/authors-charm-config.html).

I'll get something working soon.

[Charles Butler (lazypower)]

Post our conversation in a conference call the remaining action items on this ticket are:

ensuring the README clearly calls out use of https for port 8443, and updating the charm itself to not deploy with a default password.

Barring these two modifications, you will be ready for the charm store! Thank you for your continued patience during the review process David.

If you have any questions/comments/concerns about the review contact us in #juju on irc.freenode.net or email the mailing list <email address hidden>, or ask a question tagged with "juju" on http://askubuntu.com.

[David Tucker (dbtucker)]

Changed charm to deploy random password unless explicitly updated in config.yaml file by user.

Status changed to "Fix Committed"

[Charles Butler (lazypower)]

David,

thank you for the updated submission! I'm happy to see the quick turn around on this and see how close it is to being accepted to the charm store. There has been a lot of nice progress here and MapR looks like a brilliant service. With the proper steps outlined I was able to run a terrasort in about 5 minutes flat on AWS.

I ran a thorough audit on the charm, and came up with the following notes:

in the disk mounting script - hooks/lib/findMapRDisks - it makes use of the AWS metadata url. This is an infraction of the recommended quality documentation - as it binds the charm very tightly to AWS - not all cloud providers support this metadata url (see: Azure, HPCloud, Joyent)

After looking through the code, it appears the charm is only fetching the SSH Key and the TimeZone of the AZ from this metadata. Would it be out of the question to normalize the TZ's to UTC, and offer an ssh key to be provided via configuration? This would eliminate the dependency on AWS Metadata.

# Knitpicks
Some recurring behavior I noticed about the hook code was several calls to relation-set without a given relationship id, this was found in the install, config-changed, and start hooks. While its not necessarily a blocker - these should be moved to only be called from within the context of a relationship hook, such as the peer relationship hook.

And the last knitpick I have, is to rename it from maprcharm to mapr - as that reflects the service we are deploying. It will appear nice and concise to users deploying the charm if they can just type in juju deploy mapr - its already a known idiom that this is the mapr-charm.

Aside from the AWS Metadata URL I find no further issues with the charm. This is excellent work for a first revision of having MapR in the Juju charm store. In the interrim - if you'd like to see the MapR charm listed, you can push a branch of your charm to a personal namespace on launchpad

bzr push lp:~username/charms/precise/mapr/trunk

This will provide MapR being listed prior to it becoming a recommended Juju Charm.

If you have any questions/comments/concerns about the review contact us in #juju on irc.freenode.net or email the mailing list <email address hidden>, or ask a question tagged with "juju" on http://askubuntu.com.

[Launchpad Janitor (janitor)]

[Expired for Juju Charms Collection because there has been no activity for 60 days.]