Comment 3 for bug 1721416

Cassandra-log on nodec8:

INFO [Native-Transport-Requests-5] 2017-10-23 11:56:59,429 MigrationManager.java:343 - Create new table: org.apache.cassandra.config.CFMetaData@28f33c04[cfId=2cab7150-b7bb-11e7-9aa8-91d310b2446f,ksName=ContrailAnalyticsCql,cfName=objectvaluetable,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@f3d4f4e, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.Int32Type),partitionColumns=[[] | [value]],partitionKeyColumns=[key, key2],clusteringColumns=[column1],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.Int32Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[key2, value, key, column1],droppedColumns={},triggers=[],indexes=[]]
DEBUG [MigrationStage:1] 2017-10-23 11:56:59,438 ColumnFamilyStore.java:899 - Enqueuing flush of keyspaces: 0.517KiB (0%) on-heap, 0.000KiB (0%) off-heap
DEBUG [MessagingService-Outgoing-/192.168.192.6-Gossip] 2017-10-23 11:56:59,441 OutboundTcpConnection.java:495 - Unable to connect to /192.168.192.6
java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_111]
    at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_111]
    at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_111]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_111]
    at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:397) [apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:234) [apache-cassandra-3.10.jar:3.10]

Cassandra-log on nodec57:
INFO [Native-Transport-Requests-12] 2017-10-23 11:57:21,465 MigrationManager.java:343 - Create new table: org.apache.cassandra.config.CFMetaData@6f744761[cfId=39cdde90-b7bb-11e7-8acd-d7aa517683be,ksName=ContrailAnalyticsCql,cfName=objectvaluetable,flags=[COMPOUND],params=TableParams{comment=, read_repair_chance=0.0, dclocal_read_repair_chance=0.1, bloom_filter_fp_chance=0.01, crc_check_chance=1.0, gc_grace_seconds=0, default_time_to_live=0, memtable_flush_period_in_ms=0, min_index_interval=128, max_index_interval=2048, speculative_retry=99PERCENTILE, caching={'keys' : 'ALL', 'rows_per_partition' : 'NONE'}, compaction=CompactionParams{class=org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy, options={min_threshold=4, max_threshold=32}}, compression=org.apache.cassandra.schema.CompressionParams@52df9ae8, extensions={}, cdc=false},comparator=comparator(org.apache.cassandra.db.marshal.Int32Type),partitionColumns=[[] | [value]],partitionKeyColumns=[key, key2],clusteringColumns=[column1],keyValidator=org.apache.cassandra.db.marshal.CompositeType(org.apache.cassandra.db.marshal.Int32Type,org.apache.cassandra.db.marshal.UTF8Type),columnMetadata=[key2, value, key, column1],droppedColumns={},triggers=[],indexes=[]]
DEBUG [MessagingService-Outgoing-/192.168.192.6-Gossip] 2017-10-23 11:57:21,545 OutboundTcpConnection.java:495 - Unable to connect to /192.168.192.6
java.net.ConnectException: Connection refused
    at sun.nio.ch.Net.connect0(Native Method) ~[na:1.8.0_111]
    at sun.nio.ch.Net.connect(Net.java:454) ~[na:1.8.0_111]
    at sun.nio.ch.Net.connect(Net.java:446) ~[na:1.8.0_111]
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648) ~[na:1.8.0_111]
    at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:146) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnectionPool.newSocket(OutboundTcpConnectionPool.java:132) ~[apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:397) [apache-cassandra-3.10.jar:3.10]
    at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:234) [apache-cassandra-3.10.jar:3.10]

So, first nodec8 created the table objectvaluetable but due to network partition in the cluster could not propagate the schema to the other nodes, hence the cassandra-driver returned an error but Cassandra did not delete the table schema locally. Next time nodec57 created the table and again it could not propagate the schema to other nodes but kept it locally. Ultimately when the network connection was resumed, we started receiving schema-mismatch error, and collector got stuck in db_initialization.

The possible solution to the issue is to create session with only one of the Cassandra nodes for initialization, so that all the request goes to one Cassandra nodes only. once all the initialization is done, then only connect to other Cassandra nodes