I seeing this error, during RHOSP10+Contrail 4.1.1 deployment. below is the trace. 07/11/2018 10:40:52 PM [contrail-api]: SANDESH: CONNECT TO COLLECTOR: True INFO:contrail-api:SANDESH: CONNECT TO COLLECTOR: True 07/11/2018 10:40:52 PM [contrail-api]: SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE] INFO:contrail-api:SANDESH: Logging: LEVEL: [SYS_INFO] -> [SYS_NOTICE] WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: Connecting to zookeeper on 100.64.255.132:2181,100.64.255.133:2181,100.64.255.134:2181 ERROR:contrail-api:Cannot write http_port 8084 to /tmp/contrail-api.39535.http_port ERROR:contrail-api:Starting Introspect on HTTP Port 8084 ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >> ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Initializing description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector connection down >>, ] >> ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = contrail-controller-0:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 148044 http_port = 8084 start_time = 1531348852369513 collector_name = collector_ip = collector_list = [ 100.64.255.135:8086, 100.64.255.136:8086, 100.64.255.137:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >> INFO:api-0:Connecting to 100.64.255.134:2181 DEBUG:api-0:Sending request(xid=None): Connect(protocol_version=0, last_zxid_seen=0, time_out=400000, session_id=0, passwd='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', read_only=None) INFO:api-0:Zookeeper connection established, state: CONNECTED ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >> DEBUG:api-0:Sending request(xid=1): GetChildren(path='/id/virtual-networks', watcher=None) DEBUG:api-0:Received response(xid=1): [u'0000000001', u'0000000000', u'0000000002'] ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 1111 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=2): GetChildren(path='/id/security-groups/id', watcher=None) DEBUG:api-0:Received response(xid=2): [u'0000000000'] ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 678 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=3): GetData(path='/id/security-groups/id/0000000000', watcher=None) DEBUG:api-0:Received response(xid=3): ('__reserved__', ZnodeStat(czxid=4294967343, mzxid=4294967343, ctime=1531338598746, mtime=1531338598746, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=4294967343)) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = READ application = ZK response_time_in_usec = 650 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=4): Create(path='/id/security-groups/id/0000000000', data='__reserved__', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=0) DEBUG:api-0:Received error(xid=4) NodeExistsError((), {}) DEBUG:api-0:Sending request(xid=5): GetData(path='/id/security-groups/id/0000000000', watcher=None) DEBUG:api-0:Received response(xid=5): ('__reserved__', ZnodeStat(czxid=4294967343, mzxid=4294967343, ctime=1531338598746, mtime=1531338598746, version=0, cversion=0, aversion=0, ephemeralOwner=0, dataLength=12, numChildren=0, pzxid=4294967343)) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = READ application = ZK response_time_in_usec = 900 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = CREATE application = ZK response_time_in_usec = 1087 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=6): GetChildren(path='/id/tags/types', watcher=None) DEBUG:api-0:Received error(xid=6) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 633 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=7): GetChildren(path='/id/tags/values/tier', watcher=None) DEBUG:api-0:Received error(xid=7) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 637 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=8): GetChildren(path='/id/tags/values/application', watcher=None) DEBUG:api-0:Received error(xid=8) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 752 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=9): GetChildren(path='/id/tags/values/deployment', watcher=None) DEBUG:api-0:Received error(xid=9) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 649 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=10): GetChildren(path='/id/tags/values/site', watcher=None) DEBUG:api-0:Received error(xid=10) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 750 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=11): GetChildren(path='/id/tags/values/label', watcher=None) DEBUG:api-0:Received error(xid=11) NoNodeError((), {}) ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 626 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> DEBUG:api-0:Sending request(xid=12): Exists(path='/api-server-election', watcher=None) DEBUG:api-0:Received response(xid=12): ZnodeStat(czxid=4294967344, mzxid=4294967344, ctime=1531338598767, mtime=1531338598767, version=0, cversion=18168, aversion=0, ephemeralOwner=0, dataLength=0, numChildren=0, pzxid=4295017289) DEBUG:api-0:Sending request(xid=13): Create(path='/api-server-election/00158f2856144c9492567aec42438356__lock__', data='148044', acl=[ACL(perms=31, acl_list=['ALL'], id=Id(scheme='world', id='anyone'))], flags=3) DEBUG:api-0:Received response(xid=13): u'/api-server-election/00158f2856144c9492567aec42438356__lock__0000009084' DEBUG:api-0:Sending request(xid=14): GetChildren(path='/api-server-election', watcher=None) DEBUG:api-0:Received response(xid=14): [u'00158f2856144c9492567aec42438356__lock__0000009084'] WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: Connecting to database on ['100.64.255.132:9160', '100.64.255.133:9160', '100.64.255.134:9160'] ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 100.64.255.132:9160, 100.64.255.133:9160, 100.64.255.134:9160, ] status = Initializing description = >>, ] description = Collector, Database:Cassandra[] connection down >>, ] >> WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: Cassandra connection ESTABLISHED ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 100.64.255.132:9160, 100.64.255.133:9160, 100.64.255.134:9160, ] status = Up description = >>, ] description = Collector connection down >>, ] >> DEBUG:api-0:Sending request(xid=15): Delete(path=u'/api-server-election/00158f2856144c9492567aec42438356__lock__0000009084', version=-1) DEBUG:api-0:Received response(xid=15): True WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: Initializing RabbitMQ connection, urls ['pyamqp://guest:XW3Ahz9yUNVfddFNvYNaHzfHP@100.64.255.129:5672/', 'pyamqp://guest:XW3Ahz9yUNVfddFNvYNaHzfHP@100.64.255.130:5672/', 'pyamqp://guest:XW3Ahz9yUNVfddFNvYNaHzfHP@100.64.255.131:5672/'] WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection down ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 100.64.255.132:9160, 100.64.255.133:9160, 100.64.255.134:9160, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 100.64.255.129:5672, 100.64.255.130:5672, 100.64.255.131:5672, ] status = Initializing description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >> ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 100.64.255.132:9160, 100.64.255.133:9160, 100.64.255.134:9160, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 100.64.255.129:5672, 100.64.255.130:5672, 100.64.255.131:5672, ] status = Down description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >> WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiNotice: RabbitMQ connection ESTABLISHED ERROR:contrail-api:SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = contrail-controller-0 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 100.64.255.132:2181, 100.64.255.133:2181, 100.64.255.134:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Database name = Cassandra server_addrs = [ 100.64.255.132:9160, 100.64.255.133:9160, 100.64.255.134:9160, ] status = Up description = >>, << type = Database name = RabbitMQ server_addrs = [ 100.64.255.129:5672, 100.64.255.130:5672, 100.64.255.131:5672, ] status = Up description = >>, ] description = Collector connection down >>, ] >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 1100 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 885 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 806 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 793 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 694 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 989 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 655 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 513 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 626 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 604 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 618 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 607 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 661 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 568 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 597 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 608 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 683 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 726 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = SEND application = CASSANDRA response_time_in_usec = 777 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 592 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 624 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 936 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 1132 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 875 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 604 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 506 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 600 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 662 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 539 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 718 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 664 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 563 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 774 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 636 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 601 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 638 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 586 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 495 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 822 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 572 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 580 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 748 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 697 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 570 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 742 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 591 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 495 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> ERROR:contrail-api:SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = MULTIGET application = CASSANDRA response_time_in_usec = 795 response_size = 0 identifier = req-d3aef1ec-621d-4d30-a9f2-ad64964a4beb >> Python 2.7.5: /usr/bin/python Wed Jul 11 22:40:52 2018 A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. /usr/bin/contrail-api in () 6 7 if __name__ == '__main__': 8 sys.exit( 9 load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')() 10 ) load_entry_point = /usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in server_main(args_str=None) 4010 vnc_cgitb.enable(format='text') 4011 4012 main(args_str, VncApiServer(args_str)) 4013 #server_main 4014 global main = args_str = None global VncApiServer = /usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in __init__(self=, args_str='--conf_file /etc/contrail/contrail-api.conf --co...ontrail/contrail-keystone-auth.conf --worker_id 0') 1525 else: 1526 self._db_connect(self._args.reset_config) 1527 self._db_init_entries() 1528 1529 # ZK quota counter initialization self = self._db_init_entries = > /usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in _db_init_entries(self=) 2914 sc_ipam_obj = NetworkIpam('service-chain-flat-ipam', 2915 ipam_subnet_method="flat-subnet", ipam_subnets=sc_ipam_subnets) 2916 self.create_singleton_entry(sc_ipam_obj) 2917 2918 # Create pre-defined tag-type self = self.create_singleton_entry = > sc_ipam_obj = /usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in create_singleton_entry(self=, singleton_obj=, user_visible=True) 3023 # for singleton START 3024 try: 3025 cass_uuid = self._db_conn._object_db.fq_name_to_uuid(obj_type, fq_name) 3026 try: 3027 zk_uuid = self._db_conn.fq_name_to_uuid(obj_type, fq_name) cass_uuid undefined self = self._db_conn = self._db_conn._object_db = self._db_conn._object_db.fq_name_to_uuid = obj_type = 'network_ipam' fq_name = [u'default-domain', 'default-project', 'service-chain-flat-ipam'] /usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py in wrapper(*args=('network_ipam', [u'default-domain', 'default-project', 'service-chain-flat-ipam']), **kwargs={}) 512 513 self.start_time = datetime.datetime.now() 514 return func(*args, **kwargs) 515 except (AllServersUnavailable, MaximumRetryException) as e: 516 if self._conn_state != ConnectionStatus.DOWN: func = > args = ('network_ipam', [u'default-domain', 'default-project', 'service-chain-flat-ipam']) kwargs = {} /usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py in fq_name_to_uuid(self=, obj_type='network_ipam', fq_name=[u'default-domain', 'default-project', 'service-chain-flat-ipam']) 1429 raise NoIdError('%s %s' % (obj_type, fq_name_str)) 1430 if len(col_infos) > 1: 1431 raise VncError('Multi match %s for %s' % (fq_name_str, obj_type)) 1432 fq_name_uuid = col_infos.popitem()[0].split(':') 1433 if obj_type != 'route_target' and fq_name_uuid[:-1] != fq_name: global VncError = fq_name_str = u'default-domain:default-project:service-chain-flat-ipam' obj_type = 'network_ipam' : Multi match default-domain:default-project:service-chain-flat-ipam for network_ipam __class__ = __delattr__ = __dict__ = {} __doc__ = None __format__ = __getattribute__ = __getitem__ = __getslice__ = __hash__ = __init__ = __module__ = 'cfgm_common.exceptions' __new__ = __reduce__ = __reduce_ex__ = __repr__ = __setattr__ = __setstate__ = __sizeof__ = __str__ = __subclasshook__ = __unicode__ = __weakref__ = None args = (u'Multi match default-domain:default-project:service-chain-flat-ipam for network_ipam',) message = u'Multi match default-domain:default-project:service-chain-flat-ipam for network_ipam' The above is a description of an error in a Python program. Here is the original traceback: Traceback (most recent call last): File "/usr/bin/contrail-api", line 9, in load_entry_point('vnc-cfg-api-server==0.1dev', 'console_scripts', 'contrail-api')() File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 4012, in server_main main(args_str, VncApiServer(args_str)) File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1527, in __init__ self._db_init_entries() File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 2916, in _db_init_entries self.create_singleton_entry(sc_ipam_obj) File "/usr/lib/python2.7/site-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 3025, in create_singleton_entry cass_uuid = self._db_conn._object_db.fq_name_to_uuid(obj_type, fq_name) File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 514, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/cfgm_common/vnc_cassandra.py", line 1431, in fq_name_to_uuid raise VncError('Multi match %s for %s' % (fq_name_str, obj_type)) VncError: Multi match default-domain:default-project:service-chain-flat-ipam for network_ipam