Cassandra state detected DOWN on one of controller after fresh-install 2.21.1-15 from base ubuntu

[Sarath]

This is 3 controllers setup and issue seen on one of controller only after fresh-install of 2.21.1-15
This is the same image GE customer having in their setup and possibly might not seen as they might used upgrade-feature.

Topology
##########

3 controllers
8 Esx nodes

root@oblocknode04:~#
root@oblocknode04:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api active
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-discovery:0 active
contrail-schema backup
contrail-svc-monitor failed
contrail-vcenter-plugin active
ifmap active

== Contrail Database ==
supervisor-database: active
contrail-database active
contrail-database-nodemgr initializing (Cassandra state detected DOWN.)

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

>>> casandra logs

root@oblocknode04:/var/log/cassandra#
root@oblocknode04:/var/log/cassandra# tail -f system.log
 INFO [CompactionExecutor:160] 2016-01-05 19:29:36,556 CompactionTask.java (line 262) Compacted 7 sstables to [/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-13,]. 104,423,262 bytes to 105,153,495 (~100% of original) in 83,405ms = 1.202352MB/s. 160,046 total rows, 160,005 unique. Row merge counts were {1:159964, 2:41, 3:0, 4:0, 5:0, 6:0, 7:0, }
 INFO [CompactionExecutor:192] 2016-01-05 19:29:36,557 CompactionTask.java (line 105) Compacting [SSTableReader(path='/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-2-Data.db'), SSTableReader(path='/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-13-Data.db'), SSTableReader(path='/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-4-Data.db'), SSTableReader(path='/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-3-Data.db'), SSTableReader(path='/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-6-Data.db')]
 INFO [CompactionExecutor:168] 2016-01-05 19:29:39,032 CompactionTask.java (line 262) Compacted 2 sstables to []. 44,668,997 bytes to 0 (~0% of original) in 85,890ms = 0.000000MB/s. 149,999 total rows, 0 unique. Row merge counts were {1:149773, 2:113, }
 INFO [CompactionExecutor:140] 2016-01-05 19:31:38,615 CompactionTask.java (line 262) Compacted 8 sstables to [/var/lib/cassandra/data/ContrailAnalytics/StatsTableByStrTagV3/ContrailAnalytics-StatsTableByStrTagV3-ic-18,]. 155,164,728 bytes to 133,365,498 (~85% of original) in 205,495ms = 0.618931MB/s. 13,506 total rows, 12,352 unique. Row merge counts were {1:13480, 2:13, 3:0, 4:0, 5:0, 6:0, 7:0, 8:0, }
 INFO [CompactionExecutor:166] 2016-01-05 19:32:34,452 CompactionTask.java (line 262) Compacted 4 sstables to [/var/lib/cassandra/data/ContrailAnalytics/MessageTableCategory/ContrailAnalytics-MessageTableCategory-ic-12,]. 467,723,415 bytes to 109,070,527 (~23% of original) in 187,677ms = 0.554238MB/s. 93,860 total rows, 22,676 unique. Row merge counts were {1:93858, 2:1, 3:0, 4:0, }
 INFO [CompactionExecutor:162] 2016-01-05 19:32:41,372 CompactionTask.java (line 262) Compacted 4 sstables to [/var/lib/cassandra/data/ContrailAnalytics/MessageTableModuleId/ContrailAnalytics-MessageTableModuleId-ic-10,]. 481,336,923 bytes to 110,821,948 (~23% of original) in 187,860ms = 0.562589MB/s. 203,211 total rows, 48,816 unique. Row merge counts were {1:203197, 2:7, 3:0, 4:0, }
 INFO [CompactionExecutor:192] 2016-01-05 19:32:52,885 CompactionTask.java (line 262) Compacted 5 sstables to [/var/lib/cassandra/data/ContrailAnalytics/MessageTableKeyword/ContrailAnalytics-MessageTableKeyword-ic-14,]. 555,796,925 bytes to 116,861,094 (~21% of original) in 196,328ms = 0.567659MB/s. 831,748 total rows, 180,290 unique. Row merge counts were {1:831716, 2:16, 3:0, 4:0, 5:0, }
 INFO [MemoryMeter:1] 2016-01-05 19:38:50,795 Memtable.java (line 516) CFS(Keyspace='ContrailAnalytics', ColumnFamily='StatsTableByU64TagV3') liveRatio is 3.22226357215981 (just-counted was 2.9613157570530984). calculation took 24ms for 1024 columns
 INFO [MemoryMeter:1] 2016-01-05 19:41:41,936 Memtable.java (line 516) CFS(Keyspace='ContrailAnalytics', ColumnFamily='StatsTableByDblTagV3') liveRatio is 3.7144893430975277 (just-counted was 3.199275499037772). calculation took 11ms for 512 columns
 INFO [MemoryMeter:1] 2016-01-05 19:47:31,624 Memtable.java (line 516) CFS(Keyspace='DISCOVERY_SERVER', ColumnFamily='discovery') liveRatio is 1.1553879105696068 (just-counted was 1.1264850217279951). calculation took 2ms for 343 columns

root@oblocknode04:/var/log/cassandra#
root@oblocknode04:/var/log/cassandra# contrail-version
Package Version Build-ID | Repo | Package Name
-------------------------------------- ------------------------------ ----------------------------------
contrail-analytics 2.21.1-15 15
contrail-config 2.21.1-15 15
contrail-control 2.21.1-15 15
contrail-dns 2.21.1-15 15
contrail-f5 2.21.1-15 15
contrail-fabric-utils 2.21.1-15 15
contrail-install-packages 2.21.1-15~vcenter 15
contrail-install-vcenter-plugin 2.21.1-15 15
contrail-lib 2.21.1-15 15
contrail-nodemgr 2.21.1-15 15
contrail-openstack-analytics 2.21.1-15 15
contrail-openstack-control 2.21.1-15 15
contrail-openstack-database 2.21.1-15 15
contrail-setup 2.21.1-15 15
contrail-utils 2.21.1-15 15
contrail-vmware-config 2.21.1-15 15
ifmap-python-client 0.1-2 15
ifmap-server 0.3.2-1contrail1 15
python-contrail 2.21.1-15 15
root@oblocknode04:/var/log/cassandra#

[Sarath (nsarath)]

Please find below the logs of 3 Controllers & Vrouters

-bash-4.1$ pwd
/users/nsarath/PR/PR-1531379
-bash-4.1$
-bash-4.1$ ls -l
total 8423492
-rwxrwxrwx 1 nsarath test 188487680 Jan 5 20:14 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 8309555200 Jan 5 20:29 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 54835200 Jan 5 20:14 Ctrl-C-log.tar*
-rwxrwxrwx 1 nsarath test 4823040 Jan 5 20:14 Vrtr-0-log.tar*
-rwxrwxrwx 1 nsarath test 4823040 Jan 5 20:15 Vrtr-1-log.tar*
-rwxrwxrwx 1 nsarath test 5120000 Jan 5 20:15 Vrtr-2-log.tar*
-rwxrwxrwx 1 nsarath test 4823040 Jan 5 20:15 Vrtr-3-log.tar*
-rwxrwxrwx 1 nsarath test 4812800 Jan 5 20:15 Vrtr-4-log.tar*
-rwxrwxrwx 1 nsarath test 4833280 Jan 5 20:15 Vrtr-5-log.tar*
-rwxrwxrwx 1 nsarath test 4843520 Jan 5 20:15 Vrtr-7-log.tar*
-rwxrwxrwx 1 nsarath test 4833280 Jan 5 20:15 Vrtr-8-log.tar*
-bash-4.1$

[Raj Reddy (rajreddy)]

The logs indicate cassandra has taken 1/2hr to come up. functionality is not impacted since 2 other cassandra nodes were up. this is a onetime occurrence, we have keep a watch..

[Megh Bhatt (meghb)]

Ctrl-B cassandra logs start from 12/03/2015 so I'm not sure this is a fresh install.

Please confirm and close if so.

[Sarath (nsarath)]

If Ctrl logs not populated due to system issues, then i will keep tab and if not then it has to be unreproducible and never should be invalid as customer also get into this same issue.. I am Fixing the state to have it open.