ERROR [InternalResponseStage:9] 2017-11-09 15:59:40,603 MigrationTask.java:95 - Configuration exception merging remote schema
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found e0df9900-c538-11e7-a44e-879b1e3437da; expected d3ba6ca0-c538-11e7-a44e-879b1e3437da)
at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:808) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:770) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.config.Schema.updateTable(Schema.java:621) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1376) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1332) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1282) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.service.MigrationTask$1.response(MigrationTask.java:91) ~[apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.net.ResponseVerbHandler.doVerb(ResponseVerbHandler.java:53) [apache-cassandra-3.10.jar:3.10]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66) [apache-cassandra-3.10.jar:3.10]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_121]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_121]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) [apache-cassandra-3.10.jar:3.10]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
The most probable theory so far is -
first an attempt to create table was done at 2017-11-09 15:59:11,914, after that there was no activity related to this and hence driver might have returned an time-out error, Causing collector to issue Create Table again (at 2017-11-09 15:59:33,968) before the first table create could actually be stored in CFIdMap. After that, the table from the first attempt got stored in CFIdMap (at 2017-11-09 15:59:37,000) and when table from second attempt tried to store it (at 2017-11-09 15:59:40,601), we hit the schema-mismatch
Still Need to confirm that the first attempt failed due to request time-out error
INFO [Native- Transport- Requests- 1] 2017-11-09 15:59:11,914 MigrationManage r.java: 343 - Create new table: org.apache. cassandra. config. CFMetaData@ 77ba86ad[ cfId=d3ba6ca0- c538-11e7- a44e-879b1e3437 da,ksName= ContrailAnalyti csCql,cfName= messagetabletim estamp, flags=[ COMPOUND] ,params= TableParams{ comment= , read_repair_ chance= 0.0, dclocal_ read_repair_ chance= 0.1, bloom_filter_ fp_chance= 0.01, crc_check_ chance= 1.0, gc_grace_seconds=0, default_ time_to_ live=0, memtable_ flush_period_ in_ms=0, min_index_ interval= 128, max_index_ interval= 2048, speculative_ retry=99PERCENT ILE, caching={'keys' : 'ALL', 'rows_per_ partition' : 'NONE'}, compaction= CompactionParam s{class= org.apache. cassandra. db.compaction. SizeTieredCompa ctionStrategy, options= {min_threshold= 4, max_threshold=32}}, compression= org.apache. cassandra. schema. CompressionPara ms@cd64306, extensions={}, cdc=false} ,comparator= comparator( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. UUIDType) ,partitionColum ns=[[] | []],partitionKe yColumns= [key, key2],clusterin gColumns= [column1, column2] ,keyValidator= org.apache. cassandra. db.marshal. CompositeType( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. Int32Type) ,columnMetadata =[key2, key, column1, column2] ,droppedColumns ={},triggers= [],indexes= []]
// no logs for ContrailAnalyti csCql keyspace for ~20 seconds
INFO [Native- Transport- Requests- 4] 2017-11-09 15:59:33,968 MigrationManage r.java: 343 - Create new table: org.apache. cassandra. config. CFMetaData@ 6026e09d[ cfId=e0df9900- c538-11e7- a44e-879b1e3437 da,ksName= ContrailAnalyti csCql,cfName= messagetabletim estamp, flags=[ COMPOUND] ,params= TableParams{ comment= , read_repair_ chance= 0.0, dclocal_ read_repair_ chance= 0.1, bloom_filter_ fp_chance= 0.01, crc_check_ chance= 1.0, gc_grace_seconds=0, default_ time_to_ live=0, memtable_ flush_period_ in_ms=0, min_index_ interval= 128, max_index_ interval= 2048, speculative_ retry=99PERCENT ILE, caching={'keys' : 'ALL', 'rows_per_ partition' : 'NONE'}, compaction= CompactionParam s{class= org.apache. cassandra. db.compaction. SizeTieredCompa ctionStrategy, options= {min_threshold= 4, max_threshold=32}}, compression= org.apache. cassandra. schema. CompressionPara ms@cd64306, extensions={}, cdc=false} ,comparator= comparator( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. UUIDType) ,partitionColum ns=[[] | []],partitionKe yColumns= [key, key2],clusterin gColumns= [column1, column2] ,keyValidator= org.apache. cassandra. db.marshal. CompositeType( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. Int32Type) ,columnMetadata =[key2, key, column1, column2] ,droppedColumns ={},triggers= [],indexes= []]
INFO [MigrationStage:1] 2017-11-09 15:59:36,999 ColumnFamilySto re.java: 406 - Initializing ContrailAnalyti csCql.messageta bletimestamp cassandra. config. CFMetaData@ 323ed862[ cfId=d3ba6ca0- c538-11e7- a44e-879b1e3437 da,ksName= ContrailAnalyti csCql,cfName= messagetabletim estamp, flags=[ COMPOUND] ,params= TableParams{ comment= , read_repair_ chance= 0.0, dclocal_ read_repair_ chance= 0.1, bloom_filter_ fp_chance= 0.01, crc_check_ chance= 1.0, gc_grace_seconds=0, default_ time_to_ live=0, memtable_ flush_period_ in_ms=0, min_index_ interval= 128, max_index_ interval= 2048, speculative_ retry=99PERCENT ILE, caching={'keys' : 'ALL', 'rows_per_ partition' : 'NONE'}, compaction= CompactionParam s{class= org.apache. cassandra. db.compaction. SizeTieredCompa ctionStrategy, options= {max_threshold= 32, min_threshold=4}}, compression= org.apache. cassandra. schema. CompressionPara ms@cd64306, extensions={}, cdc=false} ,comparator= comparator( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. UUIDType) ,partitionColum ns=[[] | []],partitionKe yColumns= [key, key2],clusterin gColumns= [column1, column2] ,keyValidator= org.apache. cassandra. db.marshal. CompositeType( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. Int32Type) ,columnMetadata =[key2, key, column1, column2] ,droppedColumns ={},triggers= [],indexes= []] to cfIdMap
DEBUG [MigrationStage:1] 2017-11-09 15:59:37,000 Schema.java:425 - Adding org.apache.
DEBUG [InternalRespon seStage: 9] 2017-11-09 15:59:40,601 CFMetaData.java:768 - applying org.apache. cassandra. config. CFMetaData@ 72a0dd39[ cfId=e0df9900- c538-11e7- a44e-879b1e3437 da,ksName= ContrailAnalyti csCql,cfName= messagetabletim estamp, flags=[ COMPOUND] ,params= TableParams{ comment= , read_repair_ chance= 0.0, dclocal_ read_repair_ chance= 0.1, bloom_filter_ fp_chance= 0.01, crc_check_ chance= 1.0, gc_grace_seconds=0, default_ time_to_ live=0, memtable_ flush_period_ in_ms=0, min_index_ interval= 128, max_index_ interval= 2048, speculative_ retry=99PERCENT ILE, caching={'keys' : 'ALL', 'rows_per_ partition' : 'NONE'}, compaction= CompactionParam s{class= org.apache. cassandra. db.compaction. SizeTieredCompa ctionStrategy, options= {max_threshold= 32, min_threshold=4}}, compression= org.apache. cassandra. schema. CompressionPara ms@cd64306, extensions={}, cdc=false} ,comparator= comparator( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. UUIDType) ,partitionColum ns=[[] | []],partitionKe yColumns= [key, key2],clusterin gColumns= [column1, column2] ,keyValidator= org.apache. cassandra. db.marshal. CompositeType( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. Int32Type) ,columnMetadata =[key2, key, column1, column2] ,droppedColumns ={},triggers= [],indexes= []] to org.apache. cassandra. config. CFMetaData@ 323ed862[ cfId=d3ba6ca0- c538-11e7- a44e-879b1e3437 da,ksName= ContrailAnalyti csCql,cfName= messagetabletim estamp, flags=[ COMPOUND] ,params= TableParams{ comment= , read_repair_ chance= 0.0, dclocal_ read_repair_ chance= 0.1, bloom_filter_ fp_chance= 0.01, crc_check_ chance= 1.0, gc_grace_seconds=0, default_ time_to_ live=0, memtable_ flush_period_ in_ms=0, min_index_ interval= 128, max_index_ interval= 2048, speculative_ retry=99PERCENT ILE, caching={'keys' : 'ALL', 'rows_per_ partition' : 'NONE'}, compaction= CompactionParam s{class= org.apache. cassandra. db.compaction. SizeTieredCompa ctionStrategy, options= {max_threshold= 32, min_threshold=4}}, compression= org.apache. cassandra. schema. CompressionPara ms@cd64306, extensions={}, cdc=false} ,comparator= comparator( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. UUIDType) ,partitionColum ns=[[] | []],partitionKe yColumns= [key, key2],clusterin gColumns= [column1, column2] ,keyValidator= org.apache. cassandra. db.marshal. CompositeType( org.apache. cassandra. db.marshal. Int32Type, org.apache. cassandra. db.marshal. Int32Type) ,columnMetadata =[key2, key, column1, column2] ,droppedColumns ={},triggers= [],indexes= []]
ERROR [InternalRespon seStage: 9] 2017-11-09 15:59:40,603 MigrationTask. java:95 - Configuration exception merging remote schema cassandra. exceptions. ConfigurationEx ception: Column family ID mismatch (found e0df9900- c538-11e7- a44e-879b1e3437 da; expected d3ba6ca0- c538-11e7- a44e-879b1e3437 da) cassandra. config. CFMetaData. validateCompati bility( CFMetaData. java:808) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. config. CFMetaData. apply(CFMetaDat a.java: 770) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. config. Schema. updateTable( Schema. java:621) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. schema. SchemaKeyspace. updateKeyspace( SchemaKeyspace. java:1376) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. schema. SchemaKeyspace. mergeSchema( SchemaKeyspace. java:1332) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. schema. SchemaKeyspace. mergeSchemaAndA nnounceVersion( SchemaKeyspace. java:1282) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. service. MigrationTask$ 1.response( MigrationTask. java:91) ~[apache- cassandra- 3.10.jar: 3.10] cassandra. net.ResponseVer bHandler. doVerb( ResponseVerbHan dler.java: 53) [apache- cassandra- 3.10.jar: 3.10] cassandra. net.MessageDeli veryTask. run(MessageDeli veryTask. java:66) [apache- cassandra- 3.10.jar: 3.10] concurrent. Executors$ RunnableAdapter .call(Executors .java:511) [na:1.8.0_121] concurrent. FutureTask. run(FutureTask. java:266) [na:1.8.0_121] concurrent. ThreadPoolExecu tor.runWorker( ThreadPoolExecu tor.java: 1142) [na:1.8.0_121] concurrent. ThreadPoolExecu tor$Worker. run(ThreadPoolE xecutor. java:617) [na:1.8.0_121] cassandra. concurrent. NamedThreadFact ory.lambda$ threadLocalDeal locator$ 0(NamedThreadFa ctory.java: 79) [apache- cassandra- 3.10.jar: 3.10] Thread. run(Thread. java:745) ~[na:1.8.0_121]
org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at org.apache.
at java.util.
at java.util.
at java.util.
at java.util.
at org.apache.
at java.lang.
The most probable theory so far is -
first an attempt to create table was done at 2017-11-09 15:59:11,914, after that there was no activity related to this and hence driver might have returned an time-out error, Causing collector to issue Create Table again (at 2017-11-09 15:59:33,968) before the first table create could actually be stored in CFIdMap. After that, the table from the first attempt got stored in CFIdMap (at 2017-11-09 15:59:37,000) and when table from second attempt tried to store it (at 2017-11-09 15:59:40,601), we hit the schema-mismatch
Still Need to confirm that the first attempt failed due to request time-out error