Email chain below consists of all the things BU has requested:


On May 30, 2018, at 4:54 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:

Also,

Could you also confirm that there is no connectivity issues (i.e. slow interface) in the cluster? 

Could you give the output of the following as well-

ifstat 1 30 (on both database and analytics nodes)
bandwidth between database nodes and between database and analytics nodes
curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)'; sleep 1m; curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)’    (on all the analytics nodes- this is to get number of writes success/failures)

Thanks,
Miraj


On May 30, 2018, at 2:03 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:

Hi Abe,
Could you also send the output of the following commands run on all the 3 database nodes-

iostat -m 2 30
nodetool cfstats; sleep 1m; nodetool cfstats

Thanks,
Miraj

On May 24, 2018, at 5:08 PM, Ping Song <email address hidden> wrote:

Abe:
I see the commands output.
Can you send the whole /var/log/Cassandra tarball as requested?
 
tar czf Cassandra01.tgz /var/log/Cassandra
 
thanks.
 
From: HATTAR, ABRAHAM <email address hidden> 
Sent: Thursday, May 24, 2018 5:12 PM
To: Ping Song <email address hidden>; Miraj Subhashbhai Kheni <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>; MOINUDDIN, KHAJA <email address hidden>; SICURANZO, DEBORAH A <email address hidden>; KNOST, JOHN <email address hidden>; BROOKES-DANIELS, JOHN <email address hidden>; BATES, SIMON <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
 
Hi Miraj and Ping,
 
I upload everything you asked for in a compressed flle name  ftp.zip
 
Please note that a directory name GC_Files contains 3 files each belong to the following DB VM
 
db01                      gc-1519981498.log                           2.2Gb
db02                      gc-1519983054.log                           2.5GB
db03                      gc-1525161802.log                           560MB
 
regards
Abe
 
 
From: Ping Song <email address hidden> 
Sent: Wednesday, May 23, 2018 11:59 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
 
+ case.
 
From: Ping Song 
Sent: Wednesday, May 23, 2018 11:58 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
 
Abe:
Can you have to collect the below info as requested by our developer Miraj?
Appreciate your response.
 
Regards
ping
 
From: Miraj Subhashbhai Kheni 
Sent: Tuesday, May 22, 2018 7:30 PM
To: HATTAR, ABRAHAM <email address hidden>
Cc: Ping Song <email address hidden>; Sundaresan Rajangam <email address hidden>
Subject: List of things to be collected for SR P2 - 2018-0427-0265
 
Hi Abe,
Could you collect the following things from all 3 database nodes and attach it to the case.

Commands-

iostat -m 30 2

nodetool tpstats

ps aux | grep cassandra

du -h /var/lib/contrail_db/data | sort -hr | head -n 20

Top -o %MEM (take 10 snapshots)

nodetool cfstats ContrailAnalyticsCql.messagetable; sleep 1m; nodetool cfstats ContrailAnalyticsCql.messagetable

nodetool compactionstats

nodetool netstats
nodetool cfhistograms ContrailAnalyticsCql
nodetool status
free -g
 
Logs-
everything under /var/log/cassandra/
/etc/cassandra/cassandra.yaml
 
We will go over it and get back to you.
 
Thanks,
Miraj