Comment 5 for bug 1768360

Email chain below consists of all the things BU has requested:

On May 30, 2018, at 4:54 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:

Also,

Could you also confirm that there is no connectivity issues (i.e. slow interface) in the cluster?

Could you give the output of the following as well-

ifstat 1 30 (on both database and analytics nodes)
bandwidth between database nodes and between database and analytics nodes
curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)'; sleep 1m; curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)’ (on all the analytics nodes- this is to get number of writes success/failures)

Thanks,
Miraj

On May 30, 2018, at 2:03 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:

Hi Abe,
Could you also send the output of the following commands run on all the 3 database nodes-

iostat -m 2 30
nodetool cfstats; sleep 1m; nodetool cfstats

Thanks,
Miraj

On May 24, 2018, at 5:08 PM, Ping Song <email address hidden> wrote:

Abe:
I see the commands output.
Can you send the whole /var/log/Cassandra tarball as requested?

tar czf Cassandra01.tgz /var/log/Cassandra

thanks.

From: HATTAR, ABRAHAM <email address hidden>
Sent: Thursday, May 24, 2018 5:12 PM
To: Ping Song <email address hidden>; Miraj Subhashbhai Kheni <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>; MOINUDDIN, KHAJA <email address hidden>; SICURANZO, DEBORAH A <email address hidden>; KNOST, JOHN <email address hidden>; BROOKES-DANIELS, JOHN <email address hidden>; BATES, SIMON <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265

Hi Miraj and Ping,

I upload everything you asked for in a compressed flle name ftp.zip

Please note that a directory name GC_Files contains 3 files each belong to the following DB VM

db01 gc-1519981498.log 2.2Gb
db02 gc-1519983054.log 2.5GB
db03 gc-1525161802.log 560MB

regards
Abe

From: Ping Song <email address hidden>
Sent: Wednesday, May 23, 2018 11:59 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265

+ case.

From: Ping Song
Sent: Wednesday, May 23, 2018 11:58 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265

Abe:
Can you have to collect the below info as requested by our developer Miraj?
Appreciate your response.

Regards
ping

From: Miraj Subhashbhai Kheni
Sent: Tuesday, May 22, 2018 7:30 PM
To: HATTAR, ABRAHAM <email address hidden>
Cc: Ping Song <email address hidden>; Sundaresan Rajangam <email address hidden>
Subject: List of things to be collected for SR P2 - 2018-0427-0265

Hi Abe,
Could you collect the following things from all 3 database nodes and attach it to the case.

Commands-

iostat -m 30 2

nodetool tpstats

ps aux | grep cassandra

du -h /var/lib/contrail_db/data | sort -hr | head -n 20

Top -o %MEM (take 10 snapshots)

nodetool cfstats ContrailAnalyticsCql.messagetable; sleep 1m; nodetool cfstats ContrailAnalyticsCql.messagetable

nodetool compactionstats

nodetool netstats
nodetool cfhistograms ContrailAnalyticsCql
nodetool status
free -g

Logs-
everything under /var/log/cassandra/
/etc/cassandra/cassandra.yaml

We will go over it and get back to you.

Thanks,
Miraj