[R5.0 Microservices]: configdatabase_cassandra not able to start due to space issue

[Ankit Jain]

config cassandra seems to have crashed. It has happened after 5 days.

docker logs show the following error msg

# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f619dffd0ee, pid=1, tid=0x00007f6189cbb700
#
# JRE version: OpenJDK Runtime Environment (8.0_151-b12) (build 1.8.0_151-8u151-b12-1~deb9u1-b12)
# Java VM: OpenJDK 64-Bit Server VM (25.151-b12 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: //core or core.1
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
[thread 140057101973248 also had an error]
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117
INFO: exec /docker-entrypoint.sh cassandra -f
INFO: found '10.204.217.117' in local IPs ',10.204.217.117,127.0.0.1,172.17.0.1,,'
INFO: CASSANDRA_SEEDS=10.204.217.117 CASSANDRA_LISTEN_ADDRESS=10.204.217.117

[root@nodei5 ~]# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/nodei5--vg00-lv_root 129G 1.3G 122G 2% /
[root@nodei5 ~]# free -g
              total used free shared buff/cache available
Mem: 188 4 138 0 46 183
Swap: 3 0 3

full docker logs copied here:

/cs-shared/bugs/1753763

[Sahil Sabharwal (ssabharwal)]

There are no logs present here /cs-shared/bugs/1753763. Can you copy it again?

[Ankit Jain (ankitja)]

 I also copied here: Could you please take it from here:

[root@nodeh3 ~]# cd 1753763/
[root@nodeh3 1753763]# pwd
/root/1753763
[root@nodeh3 1753763]# ls
logs

[ankitja@nodem4 1753763]$ ls
logs

[Sahil Sabharwal (ssabharwal)]

It's hard to deduce anything just from cassandra logs. If you hit the issue again, please collect the contrail logs as well and assign it back to me.