Charming Hortonworks Apache Tez

[amir sanjar]

Apache Tez, a Framework for YARN-based, Data Processing Applications In Hadoop.

Apache™ Tez is an extensible framework for building YARN based, high performance batch and interactive data processing applications in Hadoop that need to handle TB to PB scale datasets. It allows projects in the Hadoop ecosystem, such as Apache Hive and Apache Pig, as well as 3rd-party software vendors to express fit-to-purpose data processing applications in a way that meets their unique demands for fast response times and extreme throughput at petabyte scale.

Why Apache Tez
Apache Tez provides a developer API and framework to write native YARN applications that bridge the spectrum of interactive and batch workloads. It allows applications to seamlessly span the scalability dimension from GB’s to PB’s of data and 10’s to 1000’s of nodes. The Apache Tez component library allows developers to use Tez to create Hadoop applications that integrate with YARN and perform well within mixed workload Hadoop clusters.

And, since Tez is extensible and embeddable, it provides the fit-to-purpose freedom to express highly optimized data processing applications, giving them an advantage over general-purpose, end-user-facing engines such as MapReduce and Spark. Finally, it offers a customizable execution architecture that allows you to express complex computations as dataflow graphs and allows for dynamic performance optimizations based on real information about the data and the resources required to process it.

[Adam Israel (aisrael)]

Hi Amir, thanks for submitting this charm for review. I've stepped through the initial review process and found that the charm is failing `charm proof`. I think most of these are fairly quick changes. I'm going to change the status to "Incomplete", and when you're ready for another review please change the status back to "New" or "Fix Committed".

W: no copyright file

Each charm must include a copyright file, with all files copyright specified.

W: README.md includes line 13 of boilerplate README.ex
W: README.md includes line 14 of boilerplate README.ex

The last few lines of the README haven't been changed from the initial template. There's also an empty #Configuration section that could be removed, if there are no configuration settings available to be made.

E: template interface names should be changed: interface-name
I: relation provides-relation has no hooks

The metadata.yaml contains the default provides: clause. If the charm isn't going to provide anything, you can remove that section.

W: Includes template test file, tests/00-autogen

This file needs to be renamed to something other than 00-autogen.

If you have any questions/comments/concerns about the review contact us in #juju on irc.freenode.net or email the mailing list <email address hidden>, or ask a question tagged with "juju" on http://askubuntu.com.

[amir sanjar (asanjar)]

thanks adam, applied recommended fixes

[Cory Johns (johnsca)]

Amir,

Thanks for the charm and subsequent fixes. Overall, the charm looks good. There are no longer any errors or warnings from `charm proof`, the README looks good, and I was able to get it deployed by following the instructions in the README.

However, there are a couple of small concerns:

* As we discussed on IRC, the 10-deploy-tez test does not work, due to the duplicate add('hadoop') and missing add('hdp-tez')

* The instructions in the README include what appears to be spurious $1 digraphs in various places. It looks like these might be left-overs from conversion from a script. These don't actually cause any issues when following the instructions manually, but they can make the instructions more confusing and should probably be removed.

[Charles Butler (lazypower)]

Amir,

Thank you for your continued dedication to making the Big Data charms of high quality for inclusion to the juju charm store.

I see you've addressed all the prior comments in the review thread. I have nothing additional to add. You can find the bugtracker for the hdp-tez charm here: https://bugs.launchpad.net/charms/trusty/+source/hdp-tez

Congratulations on another accepted charm into the big-data charmstore

If you have any questions/comments/concerns about the review contact us in #juju on irc.freenode.net or email the mailing list <email address hidden>, or ask a question tagged with "juju" on http://askubuntu.com.