[ubuntu-havana-R1.10-#34] cassandra issues on the upgraded virtual test cluster

[Prashant Shetty]

We had a single DB node running 1.05 which we upgraded to 1.1 build 34 and we then added 2 more DB nodes as per the new suggested topology. We had issues adding the new DB nodes and finally they all seemed resolved and the cluster was working for couple of days.

Now we see that the

1. disk usage on the DB nodes is very high
 root@nodei6:/home/cassandra# du -sh .
 756G .

 root@nodei7:/var/lib/cassandra# du -sh .
 436G .

 root@nodei8:/var/lib/cassandra# du -sh .
 311G .

2. on the node which had DB role before upgrading has the cassandra folder moved to /home, while the two new nodes that we added have the cassandra data folder under /var/lib

3. analytics_data_ttl is set to default which is 48hrs, how come the disk usage is so high…?

4. also why is the disk usage different on different DB nodes….?

The system is right now hosed up w/o any disk space, we need to recover this.

--
Bhushana

[Nagabhushana R (bhushana)]

we need to document the steps to add additional DB nodes and the commands to replicate the existing data across all DB nodes.

[Raj Reddy (rajreddy)]

I think this was related to logs from vrouter which were coming continuously because of a route changes which in turn were caused by misconfiguration..

The logs have been converted to traces..