New charm: Spectrum Scale Manager

[Adalberto Medeiros]

Hello Team,

Here is a new charm for IBM Spectrum Scale Manager for review.

Deployable ibm-spectrum-scale-manager charm can be found in the below repository.
Repo : https://code.launchpad.net/~ibmcharmers/charms/trusty/ibm-spectrum-scale-manager/trunk

And, its source code can be found in the below repository
Repo : https://code.launchpad.net/~ibmcharmers/charms/trusty/layer-ibm-spectrum-scale-manager/trunk

The charm store url is : https://jujucharms.com/u/ibmcharmers/ibm-spectrum-scale-manager/trusty/2

The charm uses interface gpfs and its source code can be found in the below repository:
https://code.launchpad.net/~ibmcharmers/interface-ibm-gpfs/trunk

Please note that IBM Spectrum Scale is available on both x86 and Power. Due to unavailability of ppc64le environment, we were not able to test the charm for Power. Once we have a environment available, we will verify it for Power architecture as well.

You can get the packages for testing from the sftp site : https://canonical.brickftp.com/

Thanks

[amir sanjar (asanjar)]

Thanks for all of your work putting gpfs charm together for review. While amulet test is running, from big-data prospective, do you have instructions on how to integrate gpfs (particularly this charm) with hadoop? That would be a
important add-on to README.md

[amir sanjar (asanjar)]

amulet test failed on amazon, I am sure I am missing an step :)

2015-03-12 09:46:00 Starting deployment of local

2015-03-12 09:46:00 Deploying services...

2015-03-12 09:46:01 Deploying service gpfs using local:trusty/gpfs

2015-03-12 09:46:04 Deploying service gpfs-client using local:trusty/gpfs

2015-03-12 09:56:07 The following units had errors:

   unit: gpfs/0: machine: 8 agent-state: error details: hook failed: "install"

2015-03-12 09:56:07 Deployment stopped. run time: 607.18

Traceback (most recent call last):

  File "/tmp/bundletester-elU6v6/gpfs/tests/10-deploy", line 19, in <module>

    d.setup(timeout=900)

  File "/usr/lib/python3/dist-packages/amulet/deployer.py", line 315, in setup

    ], cwd=self.deployer_dir)

  File "/usr/lib/python3.4/subprocess.py", line 557, in check_call

    raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '['juju-deployer', '-W', '-L', '-c', '/tmp/amulet-juju-deployer-yde9m32_.json', '-e', 'local', '-t', '1000', 'local']' returned non-zero exit status 1

[amir sanjar (asanjar)]

Based on your README.md information, is it safe to say due to the constrain to have gpfs-4-1-05_ubuntu.tar.gz link set and license info, automated amulet test should fail all the time?

[amir sanjar (asanjar)]

Two suggestions:
1) Add a publicly available gpfs binary for automated testing
2) check for invalid "source" at the beginning of amulet test

[Michael Chase-Salerno (mcs-chasal)]

Thanks for the review and suggestions.

Yes, as it stands you will need a copy of GPFS for anything to work here. Since GPFS is a licensed product, I don't think we can provide a public binary, but I've put some inquiries out on that.

We'll take a look at improving the error checking though.

[Michael Chase-Salerno (mcs-chasal)]

While we can't provide a publicly available gpfs binary, we can provide one to Canonical for testing purposes, just not for distribution. I know that a public version would be best, but it's not something I can get at this point.

If that's acceptable, please let me know who to get it to. Amir, I can have it sent to you if that's the most appropriate place.

[Adalberto Medeiros (adalbas)]

Fixed tests with revision 34 as suggested.

[Antonio Rosales (arosales)]

@Michael,

Thanks for comment 6. Could we confirm the requirements. Initially in the description the user would need to obtain a license key, but in comment 6 you reference the need for a binary plus a license key.

It would be much easier, infrastructure wise, if we could just set a license key, and have the binary ready in the charm. The binary wouldn't work unless a valid key was set.

Ideally the charm would provide the binary, and then we would just need to solve the issue of not exposing the license key for testing. Could you confirm a binary will be provided?

-thanks,
Antonio

[Antonio Rosales (arosales)]

Per comment 8 I am moving the status of this bug to "Incomplete." When comment 8 is addressed please move the bug status back to "Fix Committed" when you are ready for another review.

-thanks,
Antonio

[Michael Chase-Salerno (mcs-chasal)]

Thanks for the comments.

The way GPFS is packaged will not allow us to put the binary in the charm for general distribution. I've suggested that to the team, but don't know if that's a change they would be willing to make. It certainly won't be something that's done in the short term. The license is accepted at run time (the charm does this) so the access to the binary packages is controlled.

As I posted in #6, we're happy to provide a set of packages to Canonical for automated testing, they just can't be distributed within the charm, or made publicly available at this point.

I do understand that this could be an issue for "Recommended Charm Status", so if that's the case, please let me know and we can close this out. If so I will push that up to the GPFS team and encourage a change that would allow it to be revisited in the future.

Thanks,
Mike

[Antonio Rosales (arosales)]

@ Michael,

Thanks for the continued work on this charm. I have the below suggestions. The mostly pertain to how we can handle the binaries not being included in the charm. The suggestions are made from trying to keep the user experience as smooth as possible.

--- Overview --
Suggest to add what the charm does, and a link to general information about the GPFS software. For example:
http://www-01.ibm.com/support/knowledgecenter/SSFKCN/gpfs_welcome.html?lang=en
http://www-03.ibm.com/software/products/en/software
http://www-03.ibm.com/systems/storage/spectrum/scale/

Or add this information to a section named "## IBM MQ Project Information"

--- Usage ---
Please detail instructions on what a user needs to specifically do to obtain GPFS software. README currently reads:

>To install GPFS, you need to download the packages for Ubuntu and create a apt repository. To acquire and download GPFS, follow >instructions: [add link].

Suggest to detail how the users gets the GPFS binary (what URL to visit, what web workflow to follow, what points to take note of during the web workflow), and then once the user downloads the binary detail how the user places the binary into the charm so it can be deployed locally. Also note in the caveats that the charm must be deployed locally if we can not ship the binaries in the charm and license set. Are there any trail options?

---- Creating the repositories ----
In creating the repo section if you would like the commands to read as "code" you'll need to indent 4 spaces and have a new-line before and after:
  https://guides.github.com/features/mastering-markdown/

--- Configuration ---
Ensure instructions and examples to use all config options are listed as this is the only mechanism users can follow to use the config options.

For the next iterations of your charm to make the off-line process more stream lined you may also want to consider charm resources:
  http://jujuresources.readthedocs.org/en/latest/
This is currently being used in the Apache Hadoop Charms:
  http://bazaar.launchpad.net/~bigdata-dev/charms/trusty/apache-hadoop-client/trunk/view/head:/hooks/common.py

We'll coordinate with you on how we can obtain binaries for the review and charm CI going forward. I'll move the status of this bug to "Incomplete" while you address the above. Please update the status to "Fix Committed" when you are ready for another review.

-thanks,
Antonio

[Michael Chase-Salerno (mcs-chasal)]

Added the Readme changes suggested above. I have not done the changes suggested for "next iteration". If those are needed, I'll have to get the new team to take over that portion of the work.

Thanks for the review.

[Antonio Rosales (arosales)]

Update:

The charmer team is working with IBM to get binaries of GPFS so the deployment may be validated. We will update the bug comments with our findings once we are able to test the charm with the GPFS binaries.

-thanks,
Antonio

[Antonio Rosales (arosales)]

@Michael,

Thanks for the update to the readme. The setup instructions and information on how to obtain MQ are helpful. I can also clearly read your code blocks now too.

One small knit-pick would be to add some examples on setting config values in the readme or description in the config.yaml. Specifically, which ranges/values are supported, and examples on how a may update these config values.

As stated in comment 13 we are still coordinating binaries. As soon as we get those we can verify the deployment and provide more feedback.

-thanks,
Antonio

[Matt Bruzek (mbruzek)]

# gpfs charm review

Hello Michael and thank you for the submission of the GPFS charm!

When doing a review, the [Charm Store Policy](https://jujucharms.com/docs/devel/authors-charm-policy) and the [Best Practice for Charm Authors](https://jujucharms.com/docs/devel/authors-charm-best-practice) documents outline what we are looking for. Please refer to these documents if you have questions about the review.

# charm proof
The gpfs charm passes charm proof (our lint tool) with no messages! Sadly, not every charm can claim this. This shows me you have put the time in to clear up the warning and informational messages. Well done!

# icon.svg
Normally we like to avoid letters in the icon, but I recognize that IBM is your brand, and the GPFS product may not have a recognizable pictogram to use for the icon.

# metadata.yaml
This all looks great. I would suggest adding a name for the maintainer before the email. It should be in the format “Name <emailaddress>”. It is my understanding that there is a team of charmers, so allow me to suggest “IBM Juju Support <email address hidden>” or something along those lines. If anyone wanted to email the maintainer they would know it is a team and not someone named “jujusupp”.

# config.yaml
The configuration descriptions are very brief but are OK. The descriptions will show up in the Juju GUI so they often are more verbose, including valid options or more description but these look descriptive enough.

# README.md
Other reviewers have commented on the readme, some of their issues have not been fully addressed. Please revisit comment #11 there are some good points that are not in the configuration section.

The latest readme looks to be correct markdown and converts to HTML correctly. The section on configuration of apache2 is a little vague, perhaps a link to the apache docs or more description here would be nice. The deploy steps look clear and show a relation from manager to client.

The configuration section is very short. Perhaps adding a few sentences about each configuration option, what they do or why someone would want to change them would be nice to have. One could add common configuration examples and how to pass the configuration files on the deploy command.

The contact information section is also brief. This is your area to give links to the software and where to buy the product, where to get more information (docs), and where to go when they are having problems (support). I realize that some of these links are already in the text, but having the links together in the contact section would be a welcome addition.

# hooks
The hooks are written in python and look to be well written. The code is clear and easy to follow, and looks like some significant work went into this charm!

Without the binary I was unable to test the full charm functionality. Please make sure the hooks are idempotent, meaning they can be run multiple times with no errors or different output. Make sure that all configuration options are handled in config-changed (which it appears they are) so they change when and how the user expects. These are generally the things we look for when reviewing.

After con...

[Suchitra Venugopal (suchvenu)]

Hi Antonio,
  Please follow the below steps to download the evaluation copy GPFS software.

To obtain the evaluation software:
ftp to testcase.software.ibm.com as anonymous and download the file /fromibm/aix/canonical.tar

Once the base software is installed you need to install the patches included in the download.
Then to get it installed and configured you can check out the Spectrum Scale Docs
http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp?topic=/com.ibm.cluster.gpfs.doc/gpfsbooks.html
For a step by step install guide check out the quick start guides and lab exercises on the Spectrum Scale wiki
Quick Start Guides
https://www.ibm.com/developerworks/mydeveloperworks/wikis/home?lang=en#/wiki/General%20Parallel%20File%20System%20%28GPFS%29/page/Quick%20Start%20Guides

Regards,
 Suchitra

[Antonio Rosales (arosales)]

@ suchitra,

Hello, and thanks for the work on this charm. Per our discussion when you are ready for another review put the status of the bug into "Fix Committed" and that will put the bug into the review queue @ http://review.juju.solutions/ where the community can see it and take action on it.

-thanks,
Antonio

[Adam Israel (aisrael)]

Hi Suchitra,

I had the chance to review the gpfs charm today. I've primarily focused on getting the binaries installed so that we can do a functional test of the charm. Unfortunately, I ran into a few issues.

I downloaded the tarball that you provided. Here are its contents:

.
├── gpfs_4.1.0
│   └── Linux
│   └── x86_64
│   ├── gpfs.base-4.1.0-0.x86_64.rpm
│   ├── gpfs.base_4.1.0-0_amd64.deb
│   ├── gpfs.crypto-4.1.0-0.x86_64.rpm
│   ├── gpfs.crypto_4.1.0-0_amd64.deb
│   ├── gpfs.docs-4.1.0-0.noarch.rpm
│   ├── gpfs.docs_4.1.0-0_all.deb
│   ├── gpfs.ext-4.1.0-0.x86_64.rpm
│   ├── gpfs.ext_4.1.0-0_amd64.deb
│   ├── gpfs.gpl-4.1.0-0.noarch.rpm
│   ├── gpfs.gpl_4.1.0-0_all.deb
│   ├── gpfs.gskit-8.0.50-16.x86_64.rpm
│   ├── gpfs.gskit_8.0.50-16_amd64.deb
│   ├── gpfs.msg.en-us_4.1.0-0_all.deb
│   └── gpfs.msg.en_US-4.1.0-0.noarch.rpm
├── gpfs_4.1.0.7
│   └── x86_64
│   ├── README
│   ├── changelog
│   ├── gpfs.base-4.1.0-7.x86_64.update.rpm
│   ├── gpfs.base_4.1.0-7_amd64_update.deb
│   ├── gpfs.crypto-4.1.0-7.x86_64.update.rpm
│   ├── gpfs.crypto_4.1.0-7_amd64_update.deb
│   ├── gpfs.docs-4.1.0-7.noarch.rpm
│   ├── gpfs.docs_4.1.0-7_all.deb
│   ├── gpfs.ext-4.1.0-7.x86_64.update.rpm
│   ├── gpfs.ext_4.1.0-7_amd64_update.deb
│   ├── gpfs.gpl-4.1.0-7.noarch.rpm
│   ├── gpfs.gpl_4.1.0-7_all.deb
│   ├── gpfs.gskit-8.0.50-32.x86_64.rpm
│   ├── gpfs.gskit_8.0.50-32_amd64.deb
│   ├── gpfs.msg.en-us_4.1.0-7_all.deb
│   └── gpfs.msg.en_US-4.1.0-7.noarch.rpm
└── ibm_gpfs_canonical.tar

The instructions in the README do not match the contents of the tarball on several points:

- the file name is different
- the version has changed
- there is no gpfs4.1_patches directory, which the install hook expects to exist

With my gpfs_url set to "http://10.0.3.1/gpfs_4.1.0", it attempts to fetch the Packages file via:

http://10.0.3.1/gpfs_4.1.0/gpfs4.1_patches/Linux/x86_64/./Packages

The patches, however, seem to be in the gpfs_4.1.0.7 directory, and has no Linux folder.

Follow-up Actions:
- I'd like to see either a tarball that matches the instructions in the README and/or an update to the README to reflect current installation instructions
- More detail added on the steps a user must take to install and configure the repository for installation. This should be an opinionated set of instructions, from where to extract the downloaded files, such as /srv/gpfs, and what configurations need to be made to apache2 in order for those files to be served back to the charm.

It would be nice to see a more user-friendly method for installing the binaries. I know this has been discussed before, and I'm satisfied with the current method, so long as the instructions to do so are well-documented.

Thanks again for your work on getting the charm ready to go! I'll move the charm's status back to In Progress. Once you're ready, move it back to Fix Committed and we'll take another look.

[sunitha (sradhara)]

Hi,

Modified README file.

[shilpa (shilkaul)]

Hi,

Have modified the README file.To test this charm ,download the "gpfs4.1.tar" file from the sftp site.On extracting this tar, it will create folder structure like this:

gpfs4.1
            Linux->x86_64->Packages.gz
            gpfs4.1_patches->Linux-x86_64->Packages.gz

So the Base packages are under Linux->x86_64 sub directory and the updates patches are under gpfs4.1->Linux-x86_64 sub-directory.
Set the repository URL to : "http://<server-name>/debs/gpfs4.1".

Thanks
Shilpa

[Marco Ceppi (marcoceppi)]

Thanks for the update, we'll take a look at this as soon as possible!

[shilpa (shilkaul)]

Hi Team,

The Spectrum Scale was earlier called as GPFS. We have re-worked on the old GPFS charm and created new charms called as Spectrum Scale manager and Spectrum Scale Client.

The old bug for gpfs is used here to track the review for Spectrum Scale manager, so the bug description has been changed to reflect the changes for Spectrum Scale

Thanks

[shilpa (shilkaul)]

Pulling the Spectrum Scale manager charm back as we are working to incorporate some changes for the Cinder-GPFS driver requirement. Also the charm needs to be tested for Power platform as well.
Hence pulling it back for these changes

Thanks,
Shilpa