Initial commit for WebSphere Extreme Scale - Catalog Charm

[Debottam Chatterjee]

Hi All,
Here is three new charms for IBM WXS for review.

IBM WXS Catalog charm : ibm-wxs-catalog

ibm-wxs-catalog charm:

Deployable ibm-wxs-catalog layer charm can be found in the below repository.
Deployable Charm Repo :https://code.launchpad.net/~ibmcharmers/charms/trusty/ibm-wxs-catalog/trunk
And, its source code can be found in the below repository.
https://code.launchpad.net/~ibmcharmers/charms/trusty/layer-ibm-wxs-catalog/trunk

IBM WXS Catalog charm has been pushed into charm store branch : cs:~ibmcharmers/trusty/ibm-wxs-catalog-0
link : https://jujucharms.com/u/ibmcharmers/ibm-wxs-catalog/trusty/2

Interface used:

All the above charms uses interface-ibm-wxs. Here is the source code link:
https://code.launchpad.net/~ibmcharmers/charms/trusty/interface-ibm-wxs/trunk

Thanks

[Debottam Chatterjee (debottam-chatterjee)]

IBM Installation Manager package and IBM WebSphere Extreme Scale base and fix pack packages are uploaded to canonical.brickftp.com, user: ibm-charmers.

[Matt Bruzek (mbruzek)]

Thank you for the submission of this IBM WebSphere eXtreme Scale charm. Here are my review comments:

# charm proof
The charm proof tool and it reported that there are no relations for this charm.
$ charm proof
I: all charms should provide at least one thing

I would have expected that the ibmwxs charm to have a peer relation. Meaning if you deploy 2 ibmwxs charms they can relate to each other and form a larger data grid or take advantage of having many of these charms deployed in the cloud. As the subject matter expert you know how to use IBM WebSphere eXtereme Scale and how it works. There has to be other applications that can use the Juju relation to this charm.

# README.md
There are a number of issues with the readme. I use a markdown cheat sheet when writing:
https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

Things to note are that the return characters are not translated to HTML, so you have to use two spaces after a sentence to insert a line return or the statement “Deploy charms as below: juju deploy ibmwxs” appear on the same line when converted to HTML.

There is a markdown format error for the accept-ibm-wxs-license configuration option. Removing the space before the name seemed to fix the problem. The juju set commands do not show up as a code block. To make a code block you either needs four spaces or surround the code in three back ticks (```).

Normalize the use of product names such as WebSphere and Ubuntu (search and replace the other cases of the words).

Since this charm has no relations the readme should instruct users how to use the software or what you can do with the charm after it is installed.

# icon.svg
The icon does not have multiple layers. It looks to be a flat image. The icon instructions indicate that the icon must be on the pictogram layer: https://jujucharms.com/docs/devel/authors-charm-icon

We don't normally want letters in the icon because they are hard to read at small scale. If that is the only logo for this product the letters are fine. Where does the 6 come from? The version I downloaded was 8.6. Please take another look at the icon and paste the image on the Pictogram layer of the default icon.

# Deploy
This charm is still very hard to deploy. The instructions are very minimal on setting up the apache server and error prone for users. Please add as much detail as you can to this section. Including how many files a user needs to host and what example file names would be, so they know if it is right or wrong.

I was able to set up apache to host these files, but I got several things wrong along the way and would appreciate more information here.

# config-changed
After hosting the files correctly the config-changed hook failed in error. According to the config.yaml specifying the file names was optional so I didn't specify the file names. Here is how I deployed the charm:

$ cat ~/Downloads/ibmwxs_config.yaml
ibmwxs:
  # The private url of the server that has the IBM WXS files.
  url: http://192.168.122.117
  accept-ibm-wxs-license: True
  accept-ibm-im-license: True
  sha_im: f35653e20c7925d43007a1cff83623c979109ac5
  sha_base: 801bb88913e90413a271b6102d67276a9129dcf...

[sharan (saran110)]

Hi Matt,

The issues related to icon.svg and variable cfg_im_pkg_name is resolved.The config-change hook is not going into error state.

I tested the charm by giving multiple "juju-set” command and config-changed hook got invoked every time.I was not able to reproduce any failure.Can you please elaborate more on the scenario where you were not able to initiate more than one config-changed event with the “juju set” command.

Regarding the scenario where WXS can have relation with other charms like WAS , I checked with product team.According to them , WebSphere eXtreme Scale can be integrated with other products, such as WebSphere Application Server and WebSphere Application Server Community Edition. As of now we have charm available for WebSphere application Server only.WAS can be integrated with WXS server or client type only , not WXS standalone which is the one we have charmed.

 To test this relation with other charms we don't have any charm as of now in store which we can integrate with WXS standalone.In-case in future it is available ,we can test this scenario

We can have multiple WXS standalone deployments on different containers and establish a peer relation among them.To establish the peer relation we need to do some manual steps.I have updated the README with the link providing this information.

Thanks,

[Kevin W Monroe (kwmonroe)]

Hi Sharan,

Thanks for the work on getting IBM Websphere eXtreme Scale into the juju charm ecosystem. Matt and I had another look through the charm code and noticed issues with validating the checksum on binary downloads:

http://bazaar.launchpad.net/~ibmcharmers/charms/trusty/ibmwxs/trunk/view/head:/hooks/config-changed#L188
http://bazaar.launchpad.net/~ibmcharmers/charms/trusty/ibmwxs/trunk/view/head:/hooks/config-changed#L293

In the config-changed hook, the code skips any sha1sum if the config value is missing. This happens for the IM, WXS base, and WXS fixpack binaries. If the user inadvertently left the default values in config.yaml, none of these binaries would be checksum validated. This could be dangerous -- if an end user has a compromised binary, the charm should fail to install it. The charm author policy (https://jujucharms.com/docs/devel/authors-charm-policy) requires that all software payloads have their checksums validated:

----
Must verify that any software installed or utilized is verified as coming from the intended source. Any software installed from the Ubuntu archive satisfies this due to the apt sources including cryptographic signing information.
----

One way to fix this would be to disallow empty sha config values, and exit if those values have not been provided by the user. Another way would be to provide default sha1 values -- since the charm provides default filenames, the sha1sum of these files should be known and can be used as defaults.

I'm going to put this charm into "In Progress" until this sha issue is resolved, as the current implementation violates store policy. Please let us know if you have questions/concerns regarding the proposed changes, and we'll be happy to help.

[Kevin W Monroe (kwmonroe)]

Regarding comment #3, we also had some suggestions for making ibmwxs peer relations. You mentioned:

---
We can have multiple WXS standalone deployments on different containers and establish a peer relation among them.To establish the peer relation we need to do some manual steps.I have updated the README with the link providing this information.
---

I checked the link you provided in the README [1], and it seems like WXS offers easy peering with the "startOgServer.sh" script. The manual steps you referred to could be encapsulated in a peer relation so that multiple instances of WXS could call "startOgServer.sh" with the correct quorum parameters automatically. This could look something like this:

First, create a relation named "quorum". To do that, append a peer relation to the existing metadata.yaml:
peers:
  quorum:
    interface: wxs-quorum

Next, create a ./hooks/quorum-relation-changed file to handle the peer relation. This hook would use 'relation-ids' and 'relation-list' to get data from all deployed ibmwxs units, and loop over them to build a "catalogServiceEndPoints" string as documented in the knowledge center link. This string would consist of "ip:port" data from the deployed units. This could be done in bash, or python. Here's what a python function to loop over units and build this end point string might look like:

def get_wxs_servers():
    wxs_relation = json.loads(subprocess.check_output(
        ["relation-ids", "--format=json", "quorum"]))
    if not wxs_relation:
        return ""
    else:
        wxs_relation = wxs_relation[0]
    wxs = []
    units_data = subprocess.check_output(
        ['relation-list', '-r', wxs_relation, '--format=json'])
    for unit in json.loads(units_data):
        data = json.loads(subprocess.check_output(
            ['relation-get', '-r', wxs_relation, '--format=json', '-', unit]))
        if not 'port' in data:
            continue
        wxs.append("%s:%s" % (data['private-address'], data['port']))
    wxs.sort()
    return ",".join(wxs)

The charm could use the value returned from this function as the catalogServiceEndPoints parameter when calling the "startOgServer.sh" script.

This is just an example of how a peering relation might work. This becomes a much more powerful charm when peering is supported in the charm, rather than doing manual steps.

Again, please let us know if you have questions/concerns and we'd be happy to assist. Thanks again for the work on Websphere eXtreme Scale thus far!

references:
[1]: http://www-01.ibm.com/support/knowledgecenter/SSTVLU_8.6.0/com.ibm.websphere.extremescale.doc/cxscatservbp.html?lang=en

[Kevin W Monroe (kwmonroe)]

Sharan - apologies for the multiple comments, but Matt and I stumbled across one more thing during deployment of this charm.

If the required config values (such as url and license acceptance) are not set prior to deployment, the charm will execute the install, config-changed, and start hooks, but it won't actually start the service because these required values are not set.

Setting the required configuration after deployment will only execute the "config-changed" hook which installs/configures the software correctly, but "start" is never called again.

To make sure this use case works, the "config-changed" hook should explicitly call the "start" hook at the end to ensure the service is started. You may need to test to see if the service is running and stop it prior to starting at the end of "config-changed".

With this change, users that configure ibmwxs post-deployment will have the same user experience as those that configure at deployment time.

We were able to get ibmwxs started by manually running the 'start' hook after our deployment was configured. However, the README does not describe how to use the service or determine if our deployment is running correctly. We see several java processes running on the ibmwxs unit, but the charm does not open any ports. Please add a section in the README to give more details on how this service is supposed to be used.

[sharan (saran110)]

Hi Kevin,
        Thanks for your review, I'll work on the above mentioned comments.

[Kevin W Monroe (kwmonroe)]

This review has been migrated to the new queue:

https://review.jujucharms.com/reviews/41

The latest wxs-catalog charm has been imported and should be reviewed soon.