Promote apache-core-batch-processing in Charm Store

[Cory Johns]

All of the core Apache Hadoop charms and the associated bundle are ready for review. Note that most of the logic was refactored into (a branch of) charmhelpers (this branch will need to be merged into upstream in the future).

If accepted, the charms and bundles will need to be pushed to ~bigdata-charmers and the bundle to ~charmers, with the ~bigdata-charmers charm branches becoming the promulgated / authoritative branch, in the same way that the hdp-* charms are.

Bundle branch:
lp:~bigdata-dev/charms/bundles/apache-core-batch-processing/trunk

Charm branches:
lp:~bigdata-dev/charms/trusty/apache-hadoop-yarn-master/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-master/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-client/trunk

Charm Helpers branch:
lp:~bigdata-dev/charm-helpers/framework

[Kevin W Monroe (kwmonroe)]

Fwiw, I've deployed and tested these charms, framework, and bundle extensively in both Vagrant and OpenStack (c-stack). So a big +1 LGTM! I'm looking forward to seeing these promoted.

My openstack deployment from this morning, showing charm revnos for reference:

[Services]
NAME EXPOSED CHARM
client false cs:~bigdata-dev/trusty/apache-hadoop-client-48
compute-slave false cs:~bigdata-dev/trusty/apache-hadoop-compute-slave-45
hdfs-master false cs:~bigdata-dev/trusty/apache-hadoop-hdfs-master-44
secondary-namenode false cs:~bigdata-dev/trusty/apache-hadoop-hdfs-secondary-14
yarn-master false cs:~bigdata-dev/trusty/apache-hadoop-yarn-master-45

[Units]
ID STATE VERSION MACHINE PORTS PUBLIC-ADDRESS
client/0 started 1.21.3 32 10.55.60.237
compute-slave/0 started 1.21.3 33 8042/tcp,50075/tcp 10.55.61.90
compute-slave/1 started 1.21.3 34 8042/tcp,50075/tcp 10.55.61.141
compute-slave/2 started 1.21.3 35 8042/tcp,50075/tcp 10.55.61.134
hdfs-master/0 started 1.21.3 36 8020/tcp,50070/tcp 10.55.61.135
secondary-namenode/0 started 1.21.3 37 10.55.61.182
yarn-master/0 started 1.21.3 38 8032/tcp,8088/tcp,19888/tcp 10.55.61.183

[Adam Israel (aisrael)]

Hi Cory,

Thanks for all of your work on the bundle and related charms, and the refactoring of the services framework in charmhelpers.

For the bundle and charms, you have my +1. I was able to deploy them successfully, using Amazon, with no issue and the tests passed cleanly.

As for the charmhelpers branch, I have to give it a -1 at the moment. While I'm excited to see this new version of the services framework, I note a couple of issues:
- The old charmhelpers.core.services api is missing, breaking backwards compatibility. I wonder if there's a better way to handle this kind of refactoring. A versioned namespace, perhaps, or turning charmhelpers.core.services into a compatibility layer so that charms using the existing services framework "just work" and can benefit from some of the bug fixes in the new framework.
- There are a few lint errors:
    Checking for Python syntax...
    charmhelpers/contrib/bigdata/handlers/apache.py:246:13: E265 block comment should start with '# '
    charmhelpers/contrib/bigdata/handlers/apache.py:267:17: E265 block comment should start with '# '
    charmhelpers/contrib/bigdata/handlers/apache.py:380:13: E265 block comment should start with '# '
    make: *** [lint] Error 1
- Make test has several errors
- charmhelpers.core.charmframework.base and .helpers have very low test coverage. I'd love to see that expanded.

[Cory Johns (johnsca)]

Thanks.

I definitely knew that the CH branch was not in any way ready for
upstream, and I was working this week on starting to correct that
(focusing on merging upstream changes and cleaning up the docs). I'll
definitely add the lint errors and test coverage to the list of items
to get that ready.

I, too, was worried about the backwards compatibility issue. Which is
why I was working on the upgrade path document; not just to document
the steps required, but also to explore what would have to be changed
to see if a compatibility layer or such would be required. Not sure
if you had a chance to look at it [1] yet, but I'd appreciate feedback
on that as well. From that, I think a deprecated compat layer is very
do-able.

Thanks again for your review.

[1]: https://docs.google.com/a/canonical.com/document/d/1ur2b5h5RtZUW5zNhnwXKjDr8FFRu_qy-R70T3MUdbNA/edit

On Thu, Apr 9, 2015 at 11:44 AM, Adam Israel <email address hidden> wrote:
> Hi Cory,
>
> Thanks for all of your work on the bundle and related charms, and the
> refactoring of the services framework in charmhelpers.
>
> For the bundle and charms, you have my +1. I was able to deploy them
> successfully, using Amazon, with no issue and the tests passed cleanly.
>
> As for the charmhelpers branch, I have to give it a -1 at the moment. While I'm excited to see this new version of the services framework, I note a couple of issues:
> - The old charmhelpers.core.services api is missing, breaking backwards compatibility. I wonder if there's a better way to handle this kind of refactoring. A versioned namespace, perhaps, or turning charmhelpers.core.services into a compatibility layer so that charms using the existing services framework "just work" and can benefit from some of the bug fixes in the new framework.
> - There are a few lint errors:
> Checking for Python syntax...
> charmhelpers/contrib/bigdata/handlers/apache.py:246:13: E265 block comment should start with '# '
> charmhelpers/contrib/bigdata/handlers/apache.py:267:17: E265 block comment should start with '# '
> charmhelpers/contrib/bigdata/handlers/apache.py:380:13: E265 block comment should start with '# '
> make: *** [lint] Error 1
> - Make test has several errors
> - charmhelpers.core.charmframework.base and .helpers have very low test coverage. I'd love to see that expanded.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1440161
>
> Title:
> Promote apache-core-batch-processing in Charm Store
>
> Status in Juju Charms:
> New
>
> Bug description:
> All of the core Apache Hadoop charms and the associated bundle are
> ready for review. Note that most of the logic was refactored into (a
> branch of) charmhelpers (this branch will need to be merged into
> upstream in the future).
>
> If accepted, the charms and bundles will need to be pushed to
> ~bigdata-charmers and the bundle to ~charmers, with the ~bigdata-
> charmers charm branches becoming the promulgated / authoritative
> branch, in the same way that the hdp-* charms are.
>
> Bundle branch:
> lp:~bigdata-dev/charms/bundles/apache-core-batch-processin...

[Whit Morriss (whitmo)]

The migration path is tricky here. If we consider the future of charmhelpers, maybe a possible solution is to make the new version of the services framework it's own lib and deprecate but not remove the legacy copy.

The move from RelationContexts to Relation objects seems big enough to warrant a "its a new world" sort of break.

[Cory Johns (johnsca)]

I think the planned path forward is the switch over to using the reactive pattern, relation stubs, and charm generate, and to move away from the services framework entirely. The work on the new patterns is going well, but we'd like to get these promoted before then. In the meantime, we have three options:

* Keep the charms as-is, with the CH fork with the framework refactors, with the understanding that the fork will eventually die off on its own once the reactive pattern is ready

* Revert the framework refactors in the charms and go back to using the main CH branch

* Merge the CH branch to trunk, possibly with a compat layer to ease the path for the charms that are now using the framework

My preference is the first option, as it is the least work and doesn't involve spending effort on a framework that we know is going to be deprecated soon.

[Kevin W Monroe (kwmonroe)]

We've gone with the path of least resistance; that is, option 1 in cory's previous comment. The modified CH branch included in our apache charms will eventually die in the reactive refactor, but we'll keep it for use by the big data charms until that refactor is ready. This won't affect any other users of charm-helpers, as it is only available from our big data repository and not upstream.

I've pushed 8 ~bigdata-dev charms and 3 bundles over to ~bigdata-charmers. The new branches (linked to this bug) are the authoritative place for our stable big data apache offerings.

[Kevin W Monroe (kwmonroe)]

Requesting ~charmers promulgation for the newly pushed ~bigdata-charmers charms/bundles (Apache only).

[Marco Ceppi (marcoceppi)]

Bundle branch:
lp:~bigdata-dev/charms/bundles/apache-core-batch-processing/trunk

Charm branches:
lp:~bigdata-dev/charms/trusty/apache-hadoop-yarn-master/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-master/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-hdfs-secondary/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-compute-slave/trunk
lp:~bigdata-dev/charms/trusty/apache-hadoop-client/trunk

have been promulgated.

[Kevin W Monroe (kwmonroe)]

Hey Marco, many thanks for the prom, but I think you missed a couple:

lp:~bigdata-charmers/charms/trusty/apache-hadoop-plugin/trunk
lp:~bigdata-charmers/charms/bundles/apache-core-batch-processing/bundle

I say this because the other related branches in this bug have gone to "lp:charms" whereas these two remain "lp:bigdata-charmers". Would you mind promulgating these as well?