New trusty/cassandra charm

[Stuart Bishop]

This is a total rewrite of the Cassandra charm. It supports Apache Cassandra 2.0, 2.1 and DataStax Enterprise 4.6.

It is not backwards compatible with the old precise/cassandra charm that only supported earlier versions of Cassandra.

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11004-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11027-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11028-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11029-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11030-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11031-results

[Review Queue (review-queue)]

This items has failed automated testing! Results available here http://reports.vapour.ws/charm-tests/charm-bundle-test-11032-results

[Matt Bruzek (mbruzek)]

Hello Stuart,

Thank you for this proposal. I had some time to review it today and found it still not passing the automated infrastructure. I was able to deploy 3 cassandra nodes manually using the README. However, when I sshed to the machiens the cqlsh command did not work that I found in the Getting Started document.

We would really like to see the tests to pass with bundletester so our automated testing system can report any problems with the charm.

I ran bundletester with the options I believe the automated testing system runs it:

bundletester -F -e local -l DEBUG -v

I got an error that tests were not passing and captured that output in a file which is attached to this bug. I also tried to run 'make test' while in the cassandra directory and got similar errors. Please fix the code so the tests run successfully.

[Matt Bruzek (mbruzek)]

I moved the bug into in progress, once you make the changes and want another review please put this bug in "Fix committed".

[Stuart Bishop (stub)]

Thanks for looking at this.

I have pushed a fix for the cqlsh issue. The port cqlsh uses changed with Cassandra 2.1, and I was setting the Cassandra 2.1 port (native protocol) rather than the Cassandra 2.0 port (thrift). 'juju ssh cassandra/0 sudo -H cqlsh' will now do what you expect, and I will document and add a test for this at some point.

The unit tests pass fine here with trusty and standard Python 3.4 installed. The tracebacks in the test failures look like you have a different version of the system mock library installed somehow. For example, the first traceback claims that line 349 of /usr/lib/python3.4/unittest/mock.py is 'self.obj = iter(obj)', but on my install it is '_mock_return_value = DEFAULT'. Perhaps a pip install of the mock library from pypi killed it?

[Stuart Bishop (stub)]

http://reports.vapour.ws/charm-tests/charm-bundle-test-11032-results/charm/charm-testing-azure/2 (amongst other previous runs) shows that the unit tests do run with bundletester.

[Matt Bruzek (mbruzek)]

Hello Stuart,

I ran the spike branch in our automated testing infrastructure and I am still seeing errors.
Please check:
http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/23/console
(output is also attached in case the text goes away)

My interpretation of this output is the 'make lint' target failed on the build system. The Makefile that I see in this branch installs the python2 version of python-virtualenv. I believe this should be the python 3 version of virtualenv.

The mock issue is similar. Currently there is no mock package or pip installed in the Makefile. If your code depends on a specific version of mock then the Makefile should install mock as a dependency.

Thank you for working with me on this. Please contact me when you make these changes, and I can kick off another set of tests on your branch. We really need these automated tests to run successfully for Cassandra.
?field.comment=Hello Stuart,

I ran the spike branch in our automated testing infrastructure and I am still seeing errors.
Please check:
http://juju-ci.vapour.ws:8080/job/charm-bundle-test-lxc/23/console
(output is also attached in case the text goes away)

My interpretation of this output is the 'make lint' target failed on the build system. The Makefile that I see in this branch installs the python2 version of python-virtualenv. I believe this should be the python 3 version of virtualenv.

The mock issue is similar. Currently there is no mock package or pip installed in the Makefile. If your code depends on a specific version of mock then the Makefile should install mock as a dependency.

Thank you for working with me on this. Please contact me when you make these changes, and I can kick off another set of tests on your branch. We really need these automated tests to run successfully for Cassandra.

[Stuart Bishop (stub)]

The test infrastructure is broken. http://pastebin.ubuntu.com/10675221/ shows a run setting up a totally fresh trusty container and running the unittests.

At the point of the first failure, the code in this branch has done nothing except a few apt commands. It then invokes 'virtualenv --python=python3' to create a python3 virtualenv, and this fails horribly in a way I cannot fathom; it appears that it is being run with the python3 interpreter, but with the PYTHONPATH set to the python2 system libraries. /usr/bin/virtualenv should not do this, as it is a python2 application (there is no python3-virtualenv package - the single virtualenv tool supports many versions of python). I have been unable to reproduce this in a clean vm and stacking virtualenvs on virtualenvs, so I'm unsure what is going on. Either the system is screwed somehow (is there another virtualenv script in the PATH, possibly in the bundletester virtualenv?), or bundletester is doing something very, very odd with the environment.

[Stuart Bishop (stub)]

I've added some debugging output, and it looks like the test environment is quite dirty. The vm is either not being reset between test runs, or the template vm is dirty. In particular, there appear to be a few pip installed python3 packages polluting the system python3 library (memcached and sugarcrm):

DEBUG:runner:3.4.0 (default, Apr 11 2014, 13:05:11)
DEBUG:runner:[GCC 4.8.2]
DEBUG:runner:['', '/usr/local/lib/python3.4/dist-packages/python3_memcached-1.51-py3.4.egg', '/usr/local/lib/python3.4/dist-packages/sugarcrm-0.1-py3.4.egg', '/usr/lib/python3.4', '/usr/lib/python3.4/plat-x86_64-linux-gnu', '/usr/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/dist-packages', '/usr/lib/python3/dist-packages']

[Stuart Bishop (stub)]

The mock library is builtin to python2.7 and python3 btw. There is also a package on pypi providing a backport for earlier versions of Python.

[Tim Van Steenburgh (tvansteenburgh)]

@stub: Unit tests pass now that jenkins has been updated to use the charmbox docker image for isolation. The integration tests are failing though. Latest run here: http://reports.vapour.ws/charm-test-details/charm-bundle-test-parent-199. If logged in you'll see links to the console and machine logs for each run. Ignore the lxc results - that env was invalid at the time.

[Stuart Bishop (stub)]

On 27 March 2015 at 21:17, Tim Van Steenburgh <email address hidden> wrote:
> @stub: Unit tests pass now that jenkins has been updated to use the
> charmbox docker image for isolation. The integration tests are failing
> though. Latest run here: http://reports.vapour.ws/charm-test-details
> /charm-bundle-test-parent-199. If logged in you'll see links to the
> console and machine logs for each run. Ignore the lxc results - that env
> was invalid at the time.

Ta. It would be lovely if a chunk of all-machines.log or similar was
available so I could see the hook failure.

I've been seeing the install hook failing here too these last two
days, and been trying to sort it. I'd assumed it was something screwed
in my own environment as 'apt-get install python3-six' was failing due
to an out of date package database.

I'll shove this MP back into the queue once I've gotten it green on
the new jenkins setup.

--
Stuart Bishop <email address hidden>

[Tim Van Steenburgh (tvansteenburgh)]

On Friday, March 27, 2015 at 11:22 AM, Stuart Bishop wrote:
> On 27 March 2015 at 21:17, Tim Van Steenburgh <<email address hidden> (mailto:<email address hidden>)> wrote:
> > @stub: Unit tests pass now that jenkins has been updated to use the
> > charmbox docker image for isolation. The integration tests are failing
> > though. Latest run here: http://reports.vapour.ws/charm-test-details
> > /charm-bundle-test-parent-199. If logged in you'll see links to the
> > console and machine logs for each run. Ignore the lxc results - that env
> > was invalid at the time.
> >
>
>
> Ta. It would be lovely if a chunk of all-machines.log or similar was
> available so I could see the hook failure.
>
Yeah, we have that now too! If you’re logged in, you should see links to
the logs on the test results page (I do).
>
> I've been seeing the install hook failing here too these last two
> days, and been trying to sort it. I'd assumed it was something screwed
> in my own environment as 'apt-get install python3-six' was failing due
> to an out of date package database.
>
> I'll shove this MP back into the queue once I've gotten it green on
> the new jenkins setup.
>
>
> --
> Stuart Bishop <<email address hidden> (mailto:<email address hidden>)>
>
> --
> You received this bug notification because you are a member of charmers,
> which is subscribed to the bug report.
> https://bugs.launchpad.net/bugs/1419116
>
> Title:
> New trusty/cassandra charm
>
> Status in Juju Charms:
> Fix Committed
>
> Bug description:
> This is a total rewrite of the Cassandra charm. It supports Apache
> Cassandra 2.0, 2.1 and DataStax Enterprise 4.6.
>
> It is not backwards compatible with the old precise/cassandra charm
> that only supported earlier versions of Cassandra.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/charms/+bug/1419116/+subscriptions
>
>

[Stuart Bishop (stub)]

Yup, should have read the notices better.

I have been unable to get things to fail the way they do in the log, using the charmbox environment and a local provider. I did fix one bug, but that should be unrelated (charmhelpers.contrib.network.ufw.enable() was deliberately aborting since it didn't like my ipv6 setup, but provides an option around that). Most of the tests seem to be passing just fine, with the only failures this last run due to the usual ssh host key conflicts causing their random failures.

I'd love a fresh test run against the other providers, but the Jenkins system I seem to be submitting to is back to the one with ConfigParser exceptions. If someone is able to throw the branch at the newer system again I'd be grateful.

[Matt Bruzek (mbruzek)]

Hello Stuart,

I the latest code in your branch against the Jenkins system. Before doing so Tim verified that all environments should now be using charmbox to run the tests.

Here are the results:

http://reports.vapour.ws/charm-test-details/charm-bundle-test-parent-204

Login and follow the links to the logs. All the tests seem to fail on "make test" there is a lot going on in the logs so I don't know exactly where the failure is but I see 'hook failed: "install"'

Please work with me (mbruzek) or Tim (tvansteenburgh) if you have any questions or need your new branch kicked off.

[Stuart Bishop (stub)]

Joyent had the better failure, which I was able to reproduce on our Openstack. While remote access to JMX (nodetool) was fine with the local provider, it was blocked with varying degrees of success in other environments.

I've updated things to make remote nodetool calls using SSH, rather than attempt to secure remote JMX access, and are good for another test run.

[Matt Bruzek (mbruzek)]

Stuart,

Thank you for your persistence and patience with testing infrastructure. I have kicked off another round of tests manually and the results should be available at: http://reports.vapour.ws/all-bundle-and-charm-results when they are done running in at most a few hours (depending on what is already queued).

[Stuart Bishop (stub)]

The only reason I've been able to come up with for these failures is the OOM killer is shutting down Cassandra (swap is disabled entirely, per documented Cassandra best practice).

I've updated the branch to inform amulet of the necessary memory requirements and need another run with the test runner.

[Tim Van Steenburgh (tvansteenburgh)]

Just kicked off another round of tests.

[Stuart Bishop (stub)]

Ta. I think it is hitting an isolation failure now, which we have seen before (python3 interpreter with python2 libraries on the path). The previous run shows it getting past this point just fine last week, and at the failure point no charm code has been run except some apt-get's and the 'virtualenv -p python3' command which fails.

[Stuart Bishop (stub)]

Nope, must have been looking in the wrong spot. The April 9 11:54 run at http://reports.vapour.ws/all-bundle-and-charm-results has real failures, or the sort I've been unable to reproduce with the local provider and OpenStack (Cassandra starting up, but shortly after shutting itself down or being killed unexpectedly).

[Stuart Bishop (stub)]

I've added more logging, capturing the Cassandra log to try and see why it is deciding to shut down.

[Tim Van Steenburgh (tvansteenburgh)]

New round of tests started.

[Stuart Bishop (stub)]

Networking is the issue. Cassandra is attempting to bind to hookenv.unit_public_ip(), it fails and Cassandra is shuts down.

[Stuart Bishop (stub)]

We are good for another round.

Fixes include reverting to binding to the public ip address rather than all interfaces, as this breaks with Cassandra 2.1, and refactoring the tests so they don't attempt to create a keyspace straight after one with the same name is dropped.

[Tim Van Steenburgh (tvansteenburgh)]

New round of tests started.

[Stuart Bishop (stub)]

Back to tripping over the public ip address issue. The services framework defaults to opening the ports using the service definition's start parameter. Unfortunately, this happens *after* all the main actions have been run the first time, which will fail on public cloud environments when the service attempts to listen on the public ip address.

I've moved the open-ports out of the start parameter, and into the main part of the service definition, before any attempt is made to start Cassandra.

[Tim Van Steenburgh (tvansteenburgh)]

Started new round of tests.

[Stuart Bishop (stub)]

My ignorance was corrected without too much mirth, and I've made fixes. I'm now binding to the wildcard interface, and broadcasting the public ip address with Cassandra 2.1 (the option to do this does not exist in earlier versions).

Ready for the next round

[Tim Van Steenburgh (tvansteenburgh)]

Next round started.

[Stuart Bishop (stub)]

Getting there. Most of the tests passed the last round.

I've made some tweaks and would like another run.

[Tim Van Steenburgh (tvansteenburgh)]

New run started!

[Stuart Bishop (stub)]

I've squished a race condition, and am ready for another run.

Azure isn't letting me create VMs with 2GB of RAM. I can probably run in 1GB, but I will attempt to lower the constraint after I've killed the remaining races and got this green.

[Tim Van Steenburgh (tvansteenburgh)]

New run started.

[Stuart Bishop (stub)]

More stuff done, button's need a-pushing.

churn, churn, churn...

[Tim Van Steenburgh (tvansteenburgh)]

Buttons pushed, beep boop.

[Stuart Bishop (stub)]

Green!

[Tim Van Steenburgh (tvansteenburgh)]

This charm has been promulgated to the store. Thanks for all your work on this stub!