| 2018-07-10 10:53:30 |
vimal |
bug |
|
|
added bug |
| 2018-07-10 10:54:04 |
vimal |
nominated for series |
|
juniperopenstack/r5.0 |
|
| 2018-07-10 10:54:04 |
vimal |
bug task added |
|
juniperopenstack/r5.0 |
|
| 2018-07-10 10:54:16 |
vimal |
juniperopenstack/r5.0: importance |
Undecided |
Critical |
|
| 2018-07-10 10:54:21 |
vimal |
juniperopenstack/r5.0: milestone |
|
r5.0.1 |
|
| 2018-07-10 11:00:19 |
Sudheendra Rao |
juniperopenstack: assignee |
|
Sundaresan Rajangam (srajanga) |
|
| 2018-07-10 11:00:33 |
Sudheendra Rao |
juniperopenstack/r5.0: assignee |
|
Sundaresan Rajangam (srajanga) |
|
| 2018-07-10 11:01:06 |
Sudheendra Rao |
nominated for series |
|
juniperopenstack/trunk |
|
| 2018-07-10 11:01:06 |
Sudheendra Rao |
bug task added |
|
juniperopenstack/trunk |
|
| 2018-07-10 11:01:16 |
Sudheendra Rao |
juniperopenstack/trunk: milestone |
r5.0.1 |
r5.1.0 |
|
| 2018-07-10 14:40:09 |
vimal |
description |
nodemgr stays in initializing state ( Cassandra state detected DOWN.)
[root@nodem14 ~]#
[root@nodem14 ~]# contrail-status
Pod Service Original Name State Status
analytics alarm-gen contrail-analytics-alarm-gen running Up 5 hours
analytics api contrail-analytics-api running Up 5 hours
analytics collector contrail-analytics-collector running Up 5 hours
analytics nodemgr contrail-nodemgr running Up 5 hours
analytics query-engine contrail-analytics-query-engine running Up 5 hours
analytics snmp-collector contrail-analytics-snmp-collector running Up 5 hours
analytics topology contrail-analytics-topology running Up 5 hours
config api contrail-controller-config-api running Up About an hour
config cassandra contrail-external-cassandra running Up 5 hours
config device-manager contrail-controller-config-devicemgr running Up 5 hours
config nodemgr contrail-nodemgr running Up 5 hours
config rabbitmq contrail-external-rabbitmq running Up 5 hours
config schema contrail-controller-config-schema running Up 5 hours
config svc-monitor contrail-controller-config-svcmonitor running Up 5 hours
config zookeeper contrail-external-zookeeper running Up 5 hours
control control contrail-controller-control-control running Up 46 minutes
control dns contrail-controller-control-dns running Up 5 hours
control named contrail-controller-control-named running Up 5 hours
control nodemgr contrail-nodemgr running Up 5 hours
database cassandra contrail-external-cassandra running Up 5 hours
database kafka contrail-external-kafka running Up 5 hours
database nodemgr contrail-nodemgr running Up 5 hours
database zookeeper contrail-external-zookeeper running Up 5 hours
webui job contrail-controller-webui-job running Up 5 hours
webui web contrail-controller-webui-web running Up 5 hours
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics ==
snmp-collector: active
query-engine: active
api: active
alarm-gen: active
nodemgr: active
collector: active
topology: active
== Contrail config ==
api: active
zookeeper: active
svc-monitor: backup
nodemgr: active
device-manager: backup
cassandra: active
rabbitmq: active
schema: backup
== Contrail webui ==
web: active
job: active
== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: active
Below logs are seen in analytics_database_cassandra_1
INFO [Service Thread] 2018-07-10 07:10:17,733 StatusLogger.java:51 - GossipStage 1 72 17282 0 0
WARN [ScheduledTasks:2] 2018-07-10 07:10:30,933 NoSpamLogger.java:94 - Some operations timed out, details available at debug level (debug.log)
INFO [Service Thread] 2018-07-10 07:10:32,541 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
INFO [HintsDispatcher:10] 2018-07-10 07:10:46,187 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 89f61382-4bfa-4534-805b-6ddb9c5f06ba-1531200407225-1.hints to endpoint /10.204.216.95: 89f61382-4bfa-4534-805b-6ddb9c5f06ba, partially
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.96"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference-Reaper:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Jul 10, 2018 7:37:12 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Jul 10, 2018 7:39:44 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
ERROR [BatchlogTasks:1] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "GossipStage:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
ERROR [Native-Transport-Requests-18] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error: |
nodemgr stays in initializing state ( Cassandra state detected DOWN.)
version
----------
ocata-5.0-134
commands
-----------
[root@nodem14 ~]#
[root@nodem14 ~]# contrail-status
Pod Service Original Name State Status
analytics alarm-gen contrail-analytics-alarm-gen running Up 5 hours
analytics api contrail-analytics-api running Up 5 hours
analytics collector contrail-analytics-collector running Up 5 hours
analytics nodemgr contrail-nodemgr running Up 5 hours
analytics query-engine contrail-analytics-query-engine running Up 5 hours
analytics snmp-collector contrail-analytics-snmp-collector running Up 5 hours
analytics topology contrail-analytics-topology running Up 5 hours
config api contrail-controller-config-api running Up About an hour
config cassandra contrail-external-cassandra running Up 5 hours
config device-manager contrail-controller-config-devicemgr running Up 5 hours
config nodemgr contrail-nodemgr running Up 5 hours
config rabbitmq contrail-external-rabbitmq running Up 5 hours
config schema contrail-controller-config-schema running Up 5 hours
config svc-monitor contrail-controller-config-svcmonitor running Up 5 hours
config zookeeper contrail-external-zookeeper running Up 5 hours
control control contrail-controller-control-control running Up 46 minutes
control dns contrail-controller-control-dns running Up 5 hours
control named contrail-controller-control-named running Up 5 hours
control nodemgr contrail-nodemgr running Up 5 hours
database cassandra contrail-external-cassandra running Up 5 hours
database kafka contrail-external-kafka running Up 5 hours
database nodemgr contrail-nodemgr running Up 5 hours
database zookeeper contrail-external-zookeeper running Up 5 hours
webui job contrail-controller-webui-job running Up 5 hours
webui web contrail-controller-webui-web running Up 5 hours
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics ==
snmp-collector: active
query-engine: active
api: active
alarm-gen: active
nodemgr: active
collector: active
topology: active
== Contrail config ==
api: active
zookeeper: active
svc-monitor: backup
nodemgr: active
device-manager: backup
cassandra: active
rabbitmq: active
schema: backup
== Contrail webui ==
web: active
job: active
== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: active
Below logs are seen in analytics_database_cassandra_1
INFO [Service Thread] 2018-07-10 07:10:17,733 StatusLogger.java:51 - GossipStage 1 72 17282 0 0
WARN [ScheduledTasks:2] 2018-07-10 07:10:30,933 NoSpamLogger.java:94 - Some operations timed out, details available at debug level (debug.log)
INFO [Service Thread] 2018-07-10 07:10:32,541 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
INFO [HintsDispatcher:10] 2018-07-10 07:10:46,187 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 89f61382-4bfa-4534-805b-6ddb9c5f06ba-1531200407225-1.hints to endpoint /10.204.216.95: 89f61382-4bfa-4534-805b-6ddb9c5f06ba, partially
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.96"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference-Reaper:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Jul 10, 2018 7:37:12 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Jul 10, 2018 7:39:44 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
ERROR [BatchlogTasks:1] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "GossipStage:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
ERROR [Native-Transport-Requests-18] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
logs
------
/cs-shared/bugs/1780948
[vappachan@nodem3 1780948]$ ls
contrail-analytics-nodemgr.log contrail-collector.log contrail-config-nodemgr.log contrail-database-nodemgr.log logs
[ |
|
| 2018-07-10 14:46:00 |
vimal |
description |
nodemgr stays in initializing state ( Cassandra state detected DOWN.)
version
----------
ocata-5.0-134
commands
-----------
[root@nodem14 ~]#
[root@nodem14 ~]# contrail-status
Pod Service Original Name State Status
analytics alarm-gen contrail-analytics-alarm-gen running Up 5 hours
analytics api contrail-analytics-api running Up 5 hours
analytics collector contrail-analytics-collector running Up 5 hours
analytics nodemgr contrail-nodemgr running Up 5 hours
analytics query-engine contrail-analytics-query-engine running Up 5 hours
analytics snmp-collector contrail-analytics-snmp-collector running Up 5 hours
analytics topology contrail-analytics-topology running Up 5 hours
config api contrail-controller-config-api running Up About an hour
config cassandra contrail-external-cassandra running Up 5 hours
config device-manager contrail-controller-config-devicemgr running Up 5 hours
config nodemgr contrail-nodemgr running Up 5 hours
config rabbitmq contrail-external-rabbitmq running Up 5 hours
config schema contrail-controller-config-schema running Up 5 hours
config svc-monitor contrail-controller-config-svcmonitor running Up 5 hours
config zookeeper contrail-external-zookeeper running Up 5 hours
control control contrail-controller-control-control running Up 46 minutes
control dns contrail-controller-control-dns running Up 5 hours
control named contrail-controller-control-named running Up 5 hours
control nodemgr contrail-nodemgr running Up 5 hours
database cassandra contrail-external-cassandra running Up 5 hours
database kafka contrail-external-kafka running Up 5 hours
database nodemgr contrail-nodemgr running Up 5 hours
database zookeeper contrail-external-zookeeper running Up 5 hours
webui job contrail-controller-webui-job running Up 5 hours
webui web contrail-controller-webui-web running Up 5 hours
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics ==
snmp-collector: active
query-engine: active
api: active
alarm-gen: active
nodemgr: active
collector: active
topology: active
== Contrail config ==
api: active
zookeeper: active
svc-monitor: backup
nodemgr: active
device-manager: backup
cassandra: active
rabbitmq: active
schema: backup
== Contrail webui ==
web: active
job: active
== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: active
Below logs are seen in analytics_database_cassandra_1
INFO [Service Thread] 2018-07-10 07:10:17,733 StatusLogger.java:51 - GossipStage 1 72 17282 0 0
WARN [ScheduledTasks:2] 2018-07-10 07:10:30,933 NoSpamLogger.java:94 - Some operations timed out, details available at debug level (debug.log)
INFO [Service Thread] 2018-07-10 07:10:32,541 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
INFO [HintsDispatcher:10] 2018-07-10 07:10:46,187 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 89f61382-4bfa-4534-805b-6ddb9c5f06ba-1531200407225-1.hints to endpoint /10.204.216.95: 89f61382-4bfa-4534-805b-6ddb9c5f06ba, partially
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.96"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference-Reaper:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Jul 10, 2018 7:37:12 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Jul 10, 2018 7:39:44 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
ERROR [BatchlogTasks:1] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "GossipStage:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
ERROR [Native-Transport-Requests-18] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
logs
------
/cs-shared/bugs/1780948
[vappachan@nodem3 1780948]$ ls
contrail-analytics-nodemgr.log contrail-collector.log contrail-config-nodemgr.log contrail-database-nodemgr.log logs
[ |
nodemgr stays in initializing state ( Cassandra state detected DOWN.) Exception: java.lang.OutOfMemoryError is seen in analytics_database_cassandra.
version
----------
ocata-5.0-134
commands
-----------
[root@nodem14 ~]#
[root@nodem14 ~]# contrail-status
Pod Service Original Name State Status
analytics alarm-gen contrail-analytics-alarm-gen running Up 5 hours
analytics api contrail-analytics-api running Up 5 hours
analytics collector contrail-analytics-collector running Up 5 hours
analytics nodemgr contrail-nodemgr running Up 5 hours
analytics query-engine contrail-analytics-query-engine running Up 5 hours
analytics snmp-collector contrail-analytics-snmp-collector running Up 5 hours
analytics topology contrail-analytics-topology running Up 5 hours
config api contrail-controller-config-api running Up About an hour
config cassandra contrail-external-cassandra running Up 5 hours
config device-manager contrail-controller-config-devicemgr running Up 5 hours
config nodemgr contrail-nodemgr running Up 5 hours
config rabbitmq contrail-external-rabbitmq running Up 5 hours
config schema contrail-controller-config-schema running Up 5 hours
config svc-monitor contrail-controller-config-svcmonitor running Up 5 hours
config zookeeper contrail-external-zookeeper running Up 5 hours
control control contrail-controller-control-control running Up 46 minutes
control dns contrail-controller-control-dns running Up 5 hours
control named contrail-controller-control-named running Up 5 hours
control nodemgr contrail-nodemgr running Up 5 hours
database cassandra contrail-external-cassandra running Up 5 hours
database kafka contrail-external-kafka running Up 5 hours
database nodemgr contrail-nodemgr running Up 5 hours
database zookeeper contrail-external-zookeeper running Up 5 hours
webui job contrail-controller-webui-job running Up 5 hours
webui web contrail-controller-webui-web running Up 5 hours
== Contrail control ==
control: active
nodemgr: active
named: active
dns: active
== Contrail analytics ==
snmp-collector: active
query-engine: active
api: active
alarm-gen: active
nodemgr: active
collector: active
topology: active
== Contrail config ==
api: active
zookeeper: active
svc-monitor: backup
nodemgr: active
device-manager: backup
cassandra: active
rabbitmq: active
schema: backup
== Contrail webui ==
web: active
job: active
== Contrail database ==
kafka: active
nodemgr: initializing (Cassandra state detected DOWN. )
zookeeper: active
cassandra: active
Below logs are seen in analytics_database_cassandra_1
INFO [Service Thread] 2018-07-10 07:10:17,733 StatusLogger.java:51 - GossipStage 1 72 17282 0 0
WARN [ScheduledTasks:2] 2018-07-10 07:10:30,933 NoSpamLogger.java:94 - Some operations timed out, details available at debug level (debug.log)
INFO [Service Thread] 2018-07-10 07:10:32,541 StatusLogger.java:51 - SecondaryIndexManagement 0 0 0 0 0
INFO [HintsDispatcher:10] 2018-07-10 07:10:46,187 HintsDispatchExecutor.java:289 - Finished hinted handoff of file 89f61382-4bfa-4534-805b-6ddb9c5f06ba-1531200407225-1.hints to endpoint /10.204.216.95: 89f61382-4bfa-4534-805b-6ddb9c5f06ba, partially
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.96"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Reference-Reaper:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
*** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message can't create byte arrau at JPLISAgent.c line: 813
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MessagingService-Incoming-/10.204.216.95"
Jul 10, 2018 7:37:12 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Jul 10, 2018 7:39:44 AM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop
WARNING: RMI TCP Accept-7200: accept loop for ServerSocket[addr=localhost/127.0.0.1,localport=7200] throws
java.lang.OutOfMemoryError: Java heap space
ERROR [BatchlogTasks:1] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "GossipStage:1"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "RMI TCP Connection(idle)"
ERROR [Native-Transport-Requests-18] 2018-07-10 07:20:23,338 JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the error:
logs
------
/cs-shared/bugs/1780948
[vappachan@nodem3 1780948]$ ls
contrail-analytics-nodemgr.log contrail-collector.log contrail-config-nodemgr.log contrail-database-nodemgr.log logs
[ |
|
| 2018-07-10 15:45:03 |
Sundaresan Rajangam |
juniperopenstack/r5.0: assignee |
Sundaresan Rajangam (srajanga) |
Santosh Gupta (sangupta) |
|
| 2018-07-10 15:45:08 |
Sundaresan Rajangam |
juniperopenstack/trunk: assignee |
Sundaresan Rajangam (srajanga) |
Santosh Gupta (sangupta) |
|
| 2018-07-10 19:34:21 |
Sundaresan Rajangam |
juniperopenstack/r5.0: status |
New |
Invalid |
|
| 2018-07-10 19:34:24 |
Sundaresan Rajangam |
juniperopenstack/trunk: status |
New |
Invalid |
|
| 2018-07-11 14:40:37 |
Sudheendra Rao |
tags |
sanityblocker |
|
|
| 2018-07-12 03:53:27 |
vimal |
tags |
|
sanityblocker |
|
| 2018-07-12 04:35:20 |
vimal |
juniperopenstack/r5.0: status |
Invalid |
New |
|
| 2018-07-12 19:08:50 |
Santosh Gupta |
juniperopenstack/r5.0: status |
New |
Incomplete |
|
| 2018-07-12 19:09:44 |
Santosh Gupta |
juniperopenstack/r5.0: assignee |
Santosh Gupta (sangupta) |
vimal (vappachan) |
|
| 2018-07-17 00:01:46 |
Santosh Gupta |
juniperopenstack/r5.0: status |
Incomplete |
New |
|
| 2018-07-17 00:01:59 |
Santosh Gupta |
juniperopenstack/trunk: status |
Invalid |
New |
|
| 2018-07-17 00:02:15 |
Santosh Gupta |
juniperopenstack/r5.0: assignee |
vimal (vappachan) |
Andrey Pavlov (apavlov-e) |
|
| 2018-07-17 00:02:32 |
Santosh Gupta |
juniperopenstack/trunk: assignee |
Santosh Gupta (sangupta) |
Andrey Pavlov (apavlov-e) |
|
| 2018-07-17 05:14:47 |
Andrey Pavlov |
juniperopenstack/r5.0: assignee |
Andrey Pavlov (apavlov-e) |
vimal (vappachan) |
|
| 2018-07-17 05:14:54 |
Andrey Pavlov |
juniperopenstack/trunk: assignee |
Andrey Pavlov (apavlov-e) |
vimal (vappachan) |
|
| 2018-07-25 10:40:41 |
Sudheendra Rao |
tags |
sanityblocker |
sanity |
|
| 2018-08-02 05:37:10 |
Sudheendra Rao |
juniperopenstack/r5.0: status |
New |
Invalid |
|
| 2018-08-02 05:37:15 |
Sudheendra Rao |
juniperopenstack/trunk: status |
New |
Invalid |
|
