Email chain below consists of all the things BU has requested:
On May 30, 2018, at 4:54 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:
Also,
Could you also confirm that there is no connectivity issues (i.e. slow interface) in the cluster?
Could you give the output of the following as well-
ifstat 1 30 (on both database and analytics nodes)
bandwidth between database nodes and between database and analytics nodes
curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)'; sleep 1m; curl http://localhost:8089/Snh_ShowCollectorServerReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath("//table_info/list"); print etree.tostring(cql_metrics[0], pretty_print=True)’ (on all the analytics nodes- this is to get number of writes success/failures)
Thanks,
Miraj
On May 30, 2018, at 2:03 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:
Hi Abe,
Could you also send the output of the following commands run on all the 3 database nodes-
On May 24, 2018, at 5:08 PM, Ping Song <email address hidden> wrote:
Abe:
I see the commands output.
Can you send the whole /var/log/Cassandra tarball as requested?
tar czf Cassandra01.tgz /var/log/Cassandra
thanks.
From: HATTAR, ABRAHAM <email address hidden>
Sent: Thursday, May 24, 2018 5:12 PM
To: Ping Song <email address hidden>; Miraj Subhashbhai Kheni <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>; MOINUDDIN, KHAJA <email address hidden>; SICURANZO, DEBORAH A <email address hidden>; KNOST, JOHN <email address hidden>; BROOKES-DANIELS, JOHN <email address hidden>; BATES, SIMON <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
Hi Miraj and Ping,
I upload everything you asked for in a compressed flle name ftp.zip
Please note that a directory name GC_Files contains 3 files each belong to the following DB VM
From: Ping Song <email address hidden>
Sent: Wednesday, May 23, 2018 11:59 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
+ case.
From: Ping Song
Sent: Wednesday, May 23, 2018 11:58 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
Abe:
Can you have to collect the below info as requested by our developer Miraj?
Appreciate your response.
Regards
ping
From: Miraj Subhashbhai Kheni
Sent: Tuesday, May 22, 2018 7:30 PM
To: HATTAR, ABRAHAM <email address hidden>
Cc: Ping Song <email address hidden>; Sundaresan Rajangam <email address hidden>
Subject: List of things to be collected for SR P2 - 2018-0427-0265
Hi Abe,
Could you collect the following things from all 3 database nodes and attach it to the case.
Commands-
iostat -m 30 2
nodetool tpstats
ps aux | grep cassandra
du -h /var/lib/contrail_db/data | sort -hr | head -n 20
Email chain below consists of all the things BU has requested:
On May 30, 2018, at 4:54 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:
Also,
Could you also confirm that there is no connectivity issues (i.e. slow interface) in the cluster?
Could you give the output of the following as well-
ifstat 1 30 (on both database and analytics nodes) localhost: 8089/Snh_ ShowCollectorSe rverReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath( "//table_ info/list" ); print etree.tostring( cql_metrics[ 0], pretty_ print=True) '; sleep 1m; curl http:// localhost: 8089/Snh_ ShowCollectorSe rverReq? | python -c 'from lxml import etree; from sys import stdin; tree = etree.parse(stdin); cql_metrics = tree.xpath( "//table_ info/list" ); print etree.tostring( cql_metrics[ 0], pretty_print=True)’ (on all the analytics nodes- this is to get number of writes success/failures)
bandwidth between database nodes and between database and analytics nodes
curl http://
Thanks,
Miraj
On May 30, 2018, at 2:03 PM, Miraj Subhashbhai Kheni <email address hidden> wrote:
Hi Abe,
Could you also send the output of the following commands run on all the 3 database nodes-
iostat -m 2 30
nodetool cfstats; sleep 1m; nodetool cfstats
Thanks,
Miraj
On May 24, 2018, at 5:08 PM, Ping Song <email address hidden> wrote:
Abe:
I see the commands output.
Can you send the whole /var/log/Cassandra tarball as requested?
tar czf Cassandra01.tgz /var/log/Cassandra
thanks.
From: HATTAR, ABRAHAM <email address hidden>
Sent: Thursday, May 24, 2018 5:12 PM
To: Ping Song <email address hidden>; Miraj Subhashbhai Kheni <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>; MOINUDDIN, KHAJA <email address hidden>; SICURANZO, DEBORAH A <email address hidden>; KNOST, JOHN <email address hidden>; BROOKES-DANIELS, JOHN <email address hidden>; BATES, SIMON <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
Hi Miraj and Ping,
I upload everything you asked for in a compressed flle name ftp.zip
Please note that a directory name GC_Files contains 3 files each belong to the following DB VM
db01 gc-1519981498.log 2.2Gb
db02 gc-1519983054.log 2.5GB
db03 gc-1525161802.log 560MB
regards
Abe
From: Ping Song <email address hidden>
Sent: Wednesday, May 23, 2018 11:59 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>; support <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
+ case.
From: Ping Song
Sent: Wednesday, May 23, 2018 11:58 AM
To: Miraj Subhashbhai Kheni <email address hidden>; HATTAR, ABRAHAM <email address hidden>
Cc: Sundaresan Rajangam <email address hidden>
Subject: RE: List of things to be collected for SR P2 - 2018-0427-0265
Abe:
Can you have to collect the below info as requested by our developer Miraj?
Appreciate your response.
Regards
ping
From: Miraj Subhashbhai Kheni
Sent: Tuesday, May 22, 2018 7:30 PM
To: HATTAR, ABRAHAM <email address hidden>
Cc: Ping Song <email address hidden>; Sundaresan Rajangam <email address hidden>
Subject: List of things to be collected for SR P2 - 2018-0427-0265
Hi Abe,
Could you collect the following things from all 3 database nodes and attach it to the case.
Commands-
iostat -m 30 2
nodetool tpstats
ps aux | grep cassandra
du -h /var/lib/ contrail_ db/data | sort -hr | head -n 20
Top -o %MEM (take 10 snapshots)
nodetool cfstats ContrailAnalyti csCql.messageta ble; sleep 1m; nodetool cfstats ContrailAnalyti csCql.messageta ble
nodetool compactionstats
nodetool netstats csCql
nodetool cfhistograms ContrailAnalyti
nodetool status
free -g
Logs- cassandra. yaml
everything under /var/log/cassandra/
/etc/cassandra/
We will go over it and get back to you.
Thanks,
Miraj