Upgrade failing from 2.22.2-14 to 3.0.2.0-36: Cassandra failing to start

Bug #1582924 reported by chhandak
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Invalid
High
Raj Reddy
R3.0
Invalid
High
Raj Reddy

Bug Description

After upgrading from 2.22.2-14 to 3.0.2.0-36 cassandra is failing to start . We have taken care of upgrading to latest kernel during upgrade procedure

root@a5d07e09-1:~# /usr/bin/nodetool status
nodetool: Failed to connect to '127.0.0.1:7199' - ConnectException: 'Connection refused'.

== Contrail Control ==
supervisor-control: active
contrail-control initializing (Number of connections:4, Expected:5)pid 3390, uptime 1:06:09
contrail-control-nodemgr active pid 3388, uptime 1:06:09
contrail-dns active pid 3391, uptime 1:06:09
contrail-named active pid 3392, uptime 1:06:09

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api initializing (Discovery:Collector[Subscribe - Status Code 503], Discovery:AlarmGenerator[Subscribe - Status Code 503], UvePartitions:UVE
-Aggregation[None], Discovery:OpServer[Publish Error - Status Code 503] connection down)pid 15410, uptime 1:02:38
contrail-analytics-nodemgr active pid 15405, uptime 1:02:38
contrail-collector timeout pid 15408, uptime 1:02:38
contrail-query-engine timeout pid 15409, uptime 1:02:38
contrail-snmp-collector initializing (Discovery:Collector[Subscribe - Status Code 503], Collector, Discovery:ApiServer[Subscribe - Status Code 503] connection d
own)pid 15406, uptime 1:02:38
contrail-topology initializing (Discovery:Collector[Subscribe - Status Code 503], Collector connection down)pid 15407, uptime 1:02:38

== Contrail Config ==
supervisor-config: inactive
contrail-api:0 initializing
contrail-config-nodemgr inactive May 17 03:15 PM
contrail-device-manager backup pid 40667, uptime 0:00:06
contrail-discovery:0 STOPPING
contrail-schema backup pid 40669, uptime 0:00:06
contrail-svc-monitor backup pid 40670, uptime 0:00:06
ifmap inactive May 17 03:15 PM

== Contrail Web UI ==
supervisor-webui: active
contrail-webui active pid 3338, uptime 1:06:14
contrail-webui-middleware active pid 3339, uptime 1:06:14

== Contrail Database ==
contrail-database: inactive
== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr initializing (Cassandra state detected DOWN.Disk space for analytics db not retrievable.)pid 11846, uptime 1:03:53
kafka active pid 11847, uptime 1:03:53

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active pid 3337, uptime 1:06:14

Cassandra error log
----------------------
INFO [main] 2016-05-17 14:21:11,175 ColumnFamilyStore.java:361 - Initializing system.hints
ERROR [main] 2016-05-17 14:21:11,183 CassandraDaemon.java:581 - Exception encountered during startup
java.lang.RuntimeException: Incompatible SSTable found. Current version ka is unable to read file: /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ic-10. Please run upgradesstables.
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:517) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:494) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:335) ~[apache-cassandra-2.1.13.jar:2.1.13] at org.apache.cassandra.db.Keyspace.<init>(Keyspace.java:275) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:121) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.db.Keyspace.open(Keyspace.java:98) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:584) ~[apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:290) [apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:564) [apache-cassandra-2.1.13.jar:2.1.13]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:653) [apache-cassandra-2.1.13.jar:2.1.13]

chhandak (chhandak)
Changed in juniperopenstack:
importance: Undecided → High
summary: - Upgrade failing from 2.22.2-14 to 3.0.2.0-36
+ Upgrade failing from 2.22.2-14 to 3.0.2.0-36: Cassandra failing to
+ start
information type: Proprietary → Public
Changed in juniperopenstack:
assignee: nobody → Raj Reddy (rajreddy)
Jeba Paulaiyan (jebap)
tags: added: blocker
Revision history for this message
chhandak (chhandak) wrote :

/auto/cores/1582924

Revision history for this message
chhandak (chhandak) wrote :

Upgrade command was wrong.

From Rel version was given 3.0.2 which should ideally be 2.22.2

Command given : fab upgrade_contrail:3.0.2,/root/contrail-install-packages_3.0.2.0-36~liberty_all.deb

Commad should be:
fab upgrade_contrail:_2.22.2,/root/contrail-install-packages_3.0.2.0-36~liberty_all.deb

Even then canssadra got upgraded to latest version.

Jeba Paulaiyan (jebap)
Changed in juniperopenstack:
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.