cassandra doesn't start while upgrading from 3.0 to 3.2 because nodetool status doesn't work

[Guilhem Tesseyre]

I was upgrading from 3.0.3.1-7~liberty to 3.2.0.0-7~mitaka and I noticed that during the upgrade when fab tries to start cassandra it doesn't work because it's trying to get the nodetool status result and nodetool just responds with : "Cassandra 2.0 and later require Java 7u25 or later", and the fab upgrade would loop like this forever :
2016-12-02 11:22:29:572656: [root@10.87.65.20] out: Configuration file '/etc/cassandra/cassandra-env.sh'
2016-12-02 11:22:29:572852: [root@10.87.65.20] out: ==> Modified (by you or by a script) since installation.
2016-12-02 11:22:29:573010: [root@10.87.65.20] out: ==> Package distributor has shipped an updated version.
2016-12-02 11:22:29:573088: [root@10.87.65.20] out: ==> Using new file as you requested.
2016-12-02 11:22:29:573158: [root@10.87.65.20] out: Installing new version of config file /etc/cassandra/cassandra-env.sh ...
2016-12-02 11:22:29:573229: [root@10.87.65.20] out: Installing new version of config file /etc/cassandra/commitlog_archiving.properties ...
2016-12-02 11:22:29:577004: [root@10.87.65.20] out: Installing new version of config file /etc/cassandra/logback.xml ...
2016-12-02 11:22:29:584668: [root@10.87.65.20] out:
2016-12-02 11:22:29:588236: [root@10.87.65.20] out: Configuration file '/etc/cassandra/cassandra.yaml'
2016-12-02 11:22:29:588337: [root@10.87.65.20] out: ==> Modified (by you or by a script) since installation.
2016-12-02 11:22:29:588384: [root@10.87.65.20] out: ==> Package distributor has shipped an updated version.
2016-12-02 11:22:29:588424: [root@10.87.65.20] out: ==> Using new file as you requested.
2016-12-02 11:22:29:588460: [root@10.87.65.20] out: Installing new version of config file /etc/cassandra/cassandra.yaml ...
2016-12-02 11:22:29:588497: [root@10.87.65.20] out: vm.max_map_count = 1048575
2016-12-02 11:22:29:652474: [root@10.87.65.20] out: net.ipv4.tcp_keepalive_time = 300
2016-12-02 11:22:29:652588: [root@10.87.65.20] out: Processing triggers for python-support (1.0.15) ...
2016-12-02 11:22:29:716448: [root@10.87.65.20] out: Processing triggers for ureadahead (0.100.0-16) ...
2016-12-02 11:22:29:880561: [root@10.87.65.20] out: [localhost] local: service cassandra stop
2016-12-02 11:22:29:912477: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:22:29:920155: [root@10.87.65.20] out: [localhost] local: service cassandra start;sleep 5
2016-12-02 11:22:35:028712: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:22:40:369170: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:22:45:677939: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:22:50:950122: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:22:56:259409: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:23:01:531545: [root@10.87.65.20] out: [localhost] local: sleep 5
2016-12-02 11:23:06:804483: [root@10.87.65.20] out: [localhost] local: sleep 5

I edited the cassandra-env.sh file as recommended here https://github.com/scylladb/scylla-tools-java/commit/d3b08e43051a50dab4d5f4bcdbe9a78ce6f3ad74 and changed the comparator used for the JVM version

-if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" \< "25" ] ; then
+if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt 25 ] ; then

Then it passes the test and proceed with the upgrade just fine.
It looks like a Cassandra bug and we might need to use the most recent cassandra version to integrate that fix.
I attached the fab upgrade contrail logs and the cassandra system logs for more information.

Guilhem

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26972
Submitter: Raj Reddy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26973
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26974
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26972
Committed: http://github.org/Juniper/contrail-provisioning/commit/54dd1ef53b42ca1a62ad697f40a4fd7eb844c17f
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 54dd1ef53b42ca1a62ad697f40a4fd7eb844c17f
Author: Raj Reddy <email address hidden>
Date: Wed Dec 7 11:26:41 2016 -0800

The cassandra-env.sh file should also be updated during upgrade. Adding
additional check to see 'nodetool status' to check cassandra up status.

Change-Id: I33ab8c6c494db46336ef7a4efefaf402618b2d30
Closes-Bug: #1646945

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26973
Committed: http://github.org/Juniper/contrail-provisioning/commit/d2210f9e33f5b26d8508140f3d9512eba1f38be6
Submitter: Zuul (<email address hidden>)
Branch: master

commit d2210f9e33f5b26d8508140f3d9512eba1f38be6
Author: Raj Reddy <email address hidden>
Date: Wed Dec 7 11:26:41 2016 -0800

The cassandra-env.sh file should also be updated during upgrade. Adding
additional check to see 'nodetool status' to check cassandra up status.

Change-Id: I33ab8c6c494db46336ef7a4efefaf402618b2d30
Closes-Bug: #1646945

[Soumil Kulkarni (soumilk)]

For upgrade from 3.0.3 to 3.1 a work around to avoid this issue is:

Edit line number 102 in "/etc/cassandra/cassandra-env.sh" to the following.

if [ "$JVM_VERSION" \< "1.8" ] && [ "$JVM_PATCH_VERSION" -lt "25" ] ; then

(add the '-lt' tag to JVM_PATCH_VERSION instead of the one that is currently there)
-------

After this restart cassandra and things should work for you.

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26974
Committed: http://github.org/Juniper/contrail-provisioning/commit/4ef9c004c5407e6e0c26658c648c9bff63234f3d
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit 4ef9c004c5407e6e0c26658c648c9bff63234f3d
Author: Raj Reddy <email address hidden>
Date: Wed Dec 7 11:26:41 2016 -0800

The cassandra-env.sh file should also be updated during upgrade. Adding
additional check to see 'nodetool status' to check cassandra up status.

Change-Id: I33ab8c6c494db46336ef7a4efefaf402618b2d30
Closes-Bug: #1646945

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28454
Submitter: Raj Reddy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28454
Committed: http://github.org/Juniper/contrail-provisioning/commit/8bf1127df8dfea2d2c51f2c6f24baf71709c1622
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit 8bf1127df8dfea2d2c51f2c6f24baf71709c1622
Author: Raj Reddy <email address hidden>
Date: Sun Feb 5 13:07:33 2017 -0800

The cassandra-env.sh file should also be updated during upgrade.
Closes-Bug: #1646945

Change-Id: Ied4ff4aa971d6767e677f33f15956f9e3911ca2d

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28543
Submitter: Raj Reddy (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28543
Committed: http://github.org/Juniper/contrail-provisioning/commit/0b042d04b7a403f30da80cfc2a2f10422a846944
Submitter: Zuul (<email address hidden>)
Branch: R3.0.3.x

commit 0b042d04b7a403f30da80cfc2a2f10422a846944
Author: Raj Reddy <email address hidden>
Date: Sun Feb 5 13:07:33 2017 -0800

The cassandra-env.sh file should also be updated during upgrade.
Closes-Bug: #1646945

Change-Id: Ied4ff4aa971d6767e677f33f15956f9e3911ca2d
(cherry picked from commit 8bf1127df8dfea2d2c51f2c6f24baf71709c1622)