Juju Charms Collection

Bug #1272083
Comment #14

Comment 14 for bug 1272083

Revision history for this message

Xiaoming Wang (xwang2713) wrote on 2014-02-20: Re: [Bug 1272083] Re: HPCC Charm initial check-in for review

#14

Thanks for the review.

For charm repository directory structure I will fix.

For the error when install/start HPCC, It could be system resource related.
We tested hpcc charm on local, amazon, mass/virtualbox. Only see similar
error when deploy multiple hpcc instance on local provide on some systems.
If this happens when deploy first instance please provide
/var/log/HPCCSystems/myroxie/roxie.log. Last a few lines to show the error
is fine. If deploy/add multiple hpcc instances on local provider roxie
process start error most likely due to error reported on
http://askubuntu.com/questions/404969/error-net-core-wmem-default-is-an-unknown-key-on-juju-local-provider

We have two systems (ubuntu 12.04 and 13.10. Both has virtubox installed)
have this problem.
Another problem we have is deploy hpcc charm on Windows to amazon: hooks
scripts are deployed outside hooks directory.

For checksum, We have a team to maintains our HPCCSystem products portal
download site: http://hpccsystems.com. Currently only md5sum are provided.
Our products are built for each supported Ubuntu and CentOS distributions.
For example, Ubuntu 12.04 amd64, Ubuntu 13.10 amd64, etc. I did copy the
md5sum of Ubuntu 12.04 to config.yaml. But for Ubuntu 13.10 user need copy
the dm5sum from our download site to replace the value in config.yaml. I
can create sha1sum for current Ubuntu 12.04 HPCC package (4.2.0-4) but we
do update product frequently (every one or two months) so to allow user use
latest HPCC it'd better keep current dm4sum method. I will request our
portal team to use sha1sum in future release and provide sha1sum files so
we don't need hard-code any checksum string in config.yaml.

I do allow empty checksum string to skip the validation. I can enforce the
check.

Thanks

On Wed, Feb 19, 2014 at 2:26 PM, Matt Bruzek
<email address hidden>wrote:

> Hello Xiaoming,
>
> Thanks for taking the time to continually improve this charm! I took a
> look at the hpcc charm this time and here is my review.
>
> You do a really good job at handling the keys in the install hook I
> particularly like how you turned off BASH echo (start +x) when you were
> handling the keys! That is a very nice touch!
>
> I also like the way you added the checksum verification in the
> configuration file. That is a very configurable solution. There are a
> few concerns with how it was implemented:
>
> If the package-checksum is not set or of zero size the install will not
> do cryptographic verification. The install hook should fail if there
> was no checksum set. Otherwise someone could unset the value and no
> cryptographic verification would be done on the downloaded file.
>
> Also I would suggest using sha1sum rather than md5sum. The underlying
> MD5 algorithm is no longer deemed secure so I would highly suggest
> sha1sum.
>
> The hpcc charm did not deploy correctly. The start hook failed with an
> exit status of 1. I looked in the logs and found that some services FAILED.
> Here are the logs from my system that show the error:
> unit-hpcc-0: 2014-02-19 18:32:08 INFO install + set +x
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + which open-port
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install
> /var/lib/juju/tools/unit-hpcc-0/open-port
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + '[' 0 -eq 0 ']'
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + open-port 8010/TCP
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + open-port 8002/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + open-port 8015/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + open-port 9876/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install +
> JUJU_HPCC_DIR=/var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + '[' '!' -e
> /var/lib/HPCCSystems/charm ']'
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + mkdir -p
> /var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + chmod -R 777
> /var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:15 INFO juju-log envgen_signature:
> unit-hpcc-0: 2014-02-19 18:32:17 INFO start Starting mydafilesrv....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:18 INFO start Starting mydali....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:19 INFO start Starting mydfuserver....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:20 INFO start Starting myeclagent....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:22 INFO start Starting myeclccserver....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:23 INFO start Starting myeclscheduler....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:24 INFO start Starting myesp....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:30 INFO start Starting myroxie....
> #033[31m [FAILED] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:31 INFO start Starting mysasha....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start Starting mythor....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start
> *****************************************
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start Stopping entire system for a
> full restart
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start
> *****************************************
> unit-hpcc-0: 2014-02-19 18:32:56 INFO start Stopping mythor...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Stopping mysasha...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Stopping myroxie...
> #033[31m [FAILED]
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Already Stopped #033[0m
> unit-hpcc-0: 2014-02-19 18:33:00 INFO start Stopping myesp...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:00 INFO start Stopping myeclscheduler...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:01 INFO start Stopping myeclccserver...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:01 INFO start Stopping myeclagent...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:08 INFO start Stopping mydfuserver...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start Stopping mydali...
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start
> ***************************************************
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start Starting the entire System
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start
> ***************************************************
> unit-hpcc-0: 2014-02-19 18:33:24 INFO start Starting mydali....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:25 INFO start Starting mydfuserver....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:26 INFO start Starting myeclagent....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:27 INFO start Starting myeclccserver....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:29 INFO start Starting myeclscheduler....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:30 INFO start Starting myesp....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:36 INFO start Starting myroxie....
> #033[31m [FAILED] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:37 INFO start Starting mysasha....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:52 INFO start Starting mythor....
> #033[32m [ OK ] #033[0m
> unit-hpcc-0: 2014-02-19 18:33:52 ERROR juju.worker.uniter uniter.go:481
> hook failed: exit status 1
>
> One last point on the directory structure. There are too many
> directories in the charm's directory structure. When I ran the command:
> bzr branch lp:~xwang2713/charms/precise/hpcc/trunk hpcc I expected to
> see the charm base directory in the new hpcc/ directory instead I found
> hpcc/precise/hpcc/ as the charm base. You should flatten this structure
> to match other charms.
>
> Thanks again for all the work on this charm. Let us know if you have
> any other questions. For the time being I've moved this bug to
> Incomplete. When you're ready for another review please move the bug
> status to either New or "Fix Committed" to have it added back in the
> queue for review.
>
> ** Changed in: charms
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1272083
>
> Title:
> HPCC Charm initial check-in for review
>
> Status in Juju Charms:
> Incomplete
>
> Bug description:
> A new Juju Charm - hpcc is submitted for review.
> The README.md has the information for how to use the charm.
>
> For charm code:
> config.yaml: there are parameters to control which HPCCSystems use
> can install. There are other parameters for how to configure
> the cluster.
> bin/ : There are some help scripts we put here mainly for
> re-configure HPCC cluster. Default configuration may not meet user need.
> We understand it probably not recommend way for juju
> charm. But we want to give user convenient way to access the tools.
> We are open for discuss these during the review process.
> icon.svg: We haven't gotten which icon we should use. It is in our
> internal process.
>
> Currently HPCC charm doesn't have related to any other charms.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charms/+bug/1272083/+subscriptions
>

Thanks for the review.

For charm repository directory structure I will fix.

For the error when install/start HPCC, It could be system resource related.
We tested hpcc charm on local, amazon, mass/virtualbox. Only see similar
error when deploy multiple hpcc instance on local provide on some systems.
If this happens when deploy first instance please provide
/var/log/HPCCSystems/myroxie/roxie.log. Last a few lines to show the error
is fine. If deploy/add multiple hpcc instances on local provider  roxie
process start error most likely due to  error reported on
http://askubuntu.com/questions/404969/error-net-core-wmem-default-is-an-unknown-key-on-juju-local-provider

I do allow empty checksum string to skip the validation. I can enforce the
check.

Thanks

On Wed, Feb 19, 2014 at 2:26 PM, Matt Bruzek
<matthew.bruzek@canonical.com>wrote:

> Hello Xiaoming,
>
> Thanks for taking the time to continually improve this charm!  I took a
> look at the hpcc charm this time and here is my review.
>
> You do a really good job at handling the keys in the install hook I
> particularly like how you turned off BASH echo (start +x) when you were
> handling the keys!  That is a very nice touch!
>
> I also like the way you added the checksum verification in the
> configuration file.  That is a very configurable solution.  There are a
> few concerns with how it was implemented:
>
> If the package-checksum is not set or of zero size the install will not
> do cryptographic verification.  The install hook should fail if there
> was no checksum set.  Otherwise someone could unset the value and no
> cryptographic verification would be done on the downloaded file.
>
> Also I would suggest using sha1sum rather than md5sum.  The underlying
> MD5 algorithm is no longer deemed secure so I would highly suggest
> sha1sum.
>
> The hpcc charm did not deploy correctly.  The start hook failed with an
> exit status of 1.  I looked in the logs and found that some services FAILED.
> Here are the logs from my system that show the error:
> unit-hpcc-0: 2014-02-19 18:32:08 INFO install + set +x
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + which open-port
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install
> /var/lib/juju/tools/unit-hpcc-0/open-port
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + '[' 0 -eq 0 ']'
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + open-port 8010/TCP
> unit-hpcc-0: 2014-02-19 18:32:09 INFO install + open-port 8002/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + open-port 8015/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + open-port 9876/TCP
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install +
> JUJU_HPCC_DIR=/var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + '[' '!' -e
> /var/lib/HPCCSystems/charm ']'
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + mkdir -p
> /var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:10 INFO install + chmod -R 777
> /var/lib/HPCCSystems/charm
> unit-hpcc-0: 2014-02-19 18:32:15 INFO juju-log envgen_signature:
> unit-hpcc-0: 2014-02-19 18:32:17 INFO start Starting mydafilesrv....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:18 INFO start Starting mydali....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:19 INFO start Starting mydfuserver....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:20 INFO start Starting myeclagent....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:22 INFO start Starting myeclccserver....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:23 INFO start Starting myeclscheduler....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:24 INFO start Starting myesp....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:30 INFO start Starting myroxie....
>  #033[31m [FAILED]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:31 INFO start Starting mysasha....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start Starting mythor....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start
> *****************************************
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start Stopping entire system for a
> full restart
> unit-hpcc-0: 2014-02-19 18:32:46 INFO start
> *****************************************
> unit-hpcc-0: 2014-02-19 18:32:56 INFO start Stopping mythor...
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Stopping mysasha...
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Stopping myroxie...
> #033[31m [FAILED]
> unit-hpcc-0: 2014-02-19 18:32:59 INFO start Already Stopped  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:00 INFO start Stopping myesp...
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:00 INFO start Stopping myeclscheduler...
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:01 INFO start Stopping myeclccserver...
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:01 INFO start Stopping myeclagent...
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:08 INFO start Stopping mydfuserver...
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start Stopping mydali...
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start
> ***************************************************
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start Starting the entire System
> unit-hpcc-0: 2014-02-19 18:33:23 INFO start
> ***************************************************
> unit-hpcc-0: 2014-02-19 18:33:24 INFO start Starting mydali....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:25 INFO start Starting mydfuserver....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:26 INFO start Starting myeclagent....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:27 INFO start Starting myeclccserver....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:29 INFO start Starting myeclscheduler....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:30 INFO start Starting myesp....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:36 INFO start Starting myroxie....
>  #033[31m [FAILED]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:37 INFO start Starting mysasha....
>  #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:52 INFO start Starting mythor....
> #033[32m [  OK  ]  #033[0m
> unit-hpcc-0: 2014-02-19 18:33:52 ERROR juju.worker.uniter uniter.go:481
> hook failed: exit status 1
>
> One last point on the directory structure.  There are too many
> directories in the charm's directory structure.  When I ran the command:
> bzr branch  lp:~xwang2713/charms/precise/hpcc/trunk hpcc I expected to
> see the charm base directory in the new hpcc/ directory instead I found
> hpcc/precise/hpcc/ as the charm base.  You should flatten this structure
> to match other charms.
>
> Thanks again for all the work on this charm.  Let us know if you have
> any other questions. For the time being I've moved this bug to
> Incomplete. When you're ready for another review please move the bug
> status to either New or "Fix Committed" to have it added back in the
> queue for review.
>
> ** Changed in: charms
>        Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1272083
>
> Title:
>   HPCC Charm initial check-in for review
>
> Status in Juju Charms:
>   Incomplete
>
> Bug description:
>   A new Juju Charm - hpcc is submitted for review.
>   The README.md has the information for how to use the charm.
>
>   For charm code:
>     config.yaml:   there are parameters to control which HPCCSystems use
> can install. There are other parameters for how to configure
>                                the cluster.
>     bin/  : There are some help scripts we put here mainly for
> re-configure HPCC cluster. Default configuration may not meet user need.
>                 We understand it probably not recommend way for juju
> charm. But we want to give user convenient  way to access the tools.
>                 We are open for discuss these during the review process.
>     icon.svg: We haven't gotten which icon we should use. It is in our
> internal process.
>
>     Currently HPCC charm doesn't have related to any other charms.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charms/+bug/1272083/+subscriptions
>