nodemgr stays in initializing state ( Cassandra state detected DOWN.)
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R5.0 |
Invalid
|
Critical
|
vimal | |||
Trunk |
Invalid
|
Critical
|
vimal |
Bug Description
nodemgr stays in initializing state ( Cassandra state detected DOWN.) Exception: java.lang.
version
----------
ocata-5.0-134
commands
-----------
[root@nodem14 ~]#
[root@nodem14 ~]# contrail-status
Pod Service Original Name State Status
analytics alarm-gen contrail-
analytics api contrail-
analytics collector contrail-
analytics nodemgr contrail-nodemgr running Up 5 hours
analytics query-engine contrail-
analytics snmp-collector contrail-
analytics topology contrail-
config api contrail-
config cassandra contrail-
config device-manager contrail-
config nodemgr contrail-nodemgr running Up 5 hours
config rabbitmq contrail-
config schema contrail-
config svc-monitor contrail-
config zookeeper contrail-
control control contrail-
control dns contrail-
control named contrail-
control nodemgr contrail-nodemgr running Up 5 hours
database cassandra contrail-
database kafka contrail-
database nodemgr contrail-nodemgr running Up 5 hours
database zookeeper contrail-
webui job contrail-
webui web contrail-
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics ==
snmp-collector: active
query-engine: active
api: active
alarm-gen: active
nodemgr: active
collector: active
topology: active
== Contrail config ==
api: active
zookeeper: active
svc-monitor: backup
nodemgr: active
device-manager: backup
cassandra: active
rabbitmq: active
schema: backup
== Contrail webui ==
web: active
job: active
== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: active
Below logs are seen in analytics_
INFO [Service Thread] 2018-07-10 07:10:17,733 StatusLogger.
WARN [ScheduledTasks:2] 2018-07-10 07:10:30,933 NoSpamLogger.
INFO [Service Thread] 2018-07-10 07:10:32,541 StatusLogger.
INFO [HintsDispatche
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
*** java.lang.
*** java.lang.
Exception: java.lang.
Exception: java.lang.
Jul 10, 2018 7:37:12 AM sun.rmi.
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[
java.lang.
Exception: java.lang.
Jul 10, 2018 7:39:44 AM sun.rmi.
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[
java.lang.
ERROR [BatchlogTasks:1] 2018-07-10 07:20:23,338 JVMStabilityIns
java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
Exception: java.lang.
ERROR [Native-
logs
------
/cs-shared/
[vappachan@nodem3 1780948]$ ls
contrail-
[
Changed in juniperopenstack: | |
assignee: | nobody → Sundaresan Rajangam (srajanga) |
description: | updated |
description: | updated |
cassandra is started with Xms 1g and Xmx 2g
This is incorrect. Xms and Xmx value should be 8g
cassand+ 1 0 99 04:57 ? 1-00:25:02 java -Xloggc: /var/log/ cassandra/ gc.log -ea -XX:+UseThreadP riorities -XX:ThreadPrior ityPolicy= 42 -XX:+HeapDumpOn OutOfMemoryErro r -Xss256k -XX:StringTable Size=1000003 -XX:+AlwaysPreTouch -XX:-UseBiasedL ocking -XX:+UseTLAB -XX:+ResizeTLAB -XX:+UseNUMA -XX:+PerfDisabl eSharedMem -Djava. net.preferIPv4S tack=true -XX:+UseParNewGC -XX:+UseConcMar kSweepGC -XX:+CMSParalle lRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuring Threshold= 1 -XX:CMSInitiati ngOccupancyFrac tion=75 -XX:+UseCMSInit iatingOccupancy Only -XX:CMSWaitDura tion=10000 -XX:+CMSParalle lInitialMarkEna bled -XX:+CMSEdenChu nksRecordAlways -XX:+CMSClassUn loadingEnabled -XX:+PrintGCDetails -XX:+PrintGCDat eStamps -XX:+PrintHeapAtGC -XX:+PrintTenur ingDistribution -XX:+PrintGCApp licationStopped Time -XX:+PrintPromo tionFailure -XX:+UseGCLogFi leRotation -XX:NumberOfGCL ogFiles= 10 -XX:GCLogFileSi ze=10M -Xms8192M -Xmx8192M -Xmn2048M -XX:CompileComm andFile= /etc/cassandra/ hotspot_ compiler -javaagent: /usr/share/ cassandra/ lib/jamm- 0.3.0.jar -Dcassandra. jmx.local. port=7199 -Dcom.sun. management. jmxremote. authenticate= false -Dcom.sun. management. jmxremote. password. file=/etc/ cassandra/ jmxremote. password -Djava. library. path=/usr/ share/cassandra /lib/sigar- bin -Xms1g -Xmx2g -Dcassandra. rpc_port= 9160 -Dcassandra. native_ transport_ port=9042 -Dcassandra. ssl_storage_ port=7011 -Dcassandra. storage_ port=7010 -Dcassandra. jmx.local. port=7200 -Dcassandra. libjemalloc= /usr/lib/ x86_64- linux-gnu/ libjemalloc. so.1 -XX:OnOutOfMemo ryError= kill -9 %p -Dlogback. configurationFi le=logback. xml -Dcassandra. logdir= /var/log/ cassandra -Dcassandra. storagedir= /var/lib/ cassandra -Dcassandra- foreground= yes -cp /etc/cassandra: /usr/share/ cassandra/ lib/HdrHistogra m-2.1.9. jar:/usr/ share/cassandra /lib/ST4- 4.0.8.jar: /usr/share/ cassandra/ lib/airline- 0.6.jar: /usr/share/ cassandra/ lib/antlr- runtime- 3.5.2.jar: /usr/share/ cassandra/ lib/asm- 5.0.4.jar: /usr/share/ cassandra/ lib/caffeine- 2.2.6.jar: /usr/share/ cassandra/ lib/cassandra- driver- core-3. 0.1-shaded. jar:/usr/ share/cassandra /lib/commons- cli-1.1. jar:/usr/ share/cassandra /lib/commons- codec-1. 9.jar:/ usr/share/ cassandra/ lib/commons- lang3-3. 1.jar:/ usr/share/ cassandra/ lib/commons- math3-3. 2.jar:/ usr/share/ cassandra/ lib/compress- lzf-0.8. 4.jar:/ usr/share/ cassandra/ lib/concurrent- trees-2. 4.0.jar: /usr/share/ cassandra/ lib/concurrentl inkedhashmap- lru-1.4. jar:/usr/ share/cassandra /lib/disruptor- 3.0.1.jar: /usr/share/ cassandra/ lib/ecj- 4.4.2.jar: /usr/share/ cassandra/ lib/guava- 18.0.jar: /usr/share/ cassandra/ lib/high- scale-lib- 1.0.6.jar: /usr/share/ cassandra/ lib/hppc- 0.5.4.jar: /usr/share/ cassandra/ lib/jackson- core-asl- 1.9.13. jar:/usr/ share/cassandra /lib/jackson- mapper- asl-1.9. 13.jar: /usr/share/ cassandra/ lib/jamm- 0.3.0.jar: /usr/share/ cassandra/ lib/javax. inject. jar:/usr/ share/cassandra /lib/jbcrypt- 0.3m.jar: /usr/share/ cassandra/ lib/jcl- over-slf4j- 1.7.7.jar: /usr/share/ cassandra/ lib/jctools- core-1. 2.1.jar: /usr/share/ cassandra/ lib/jflex- 1.6.0.jar: /usr/share/ cassandra/ lib/jna- 4.2.2.jar: /usr/share/ cassandra/ lib/joda- time-2. 4.jar:/ usr/share/ cassandra/ lib/json- simple- 1.1.jar: /usr/share/ cassandra/ lib/jsta. ..