CB r4.1-8 newton:
== Contrail Analytics ==
contrail-collector: initializing (Database:server6:Global connection down)
contrail-analytics-api: initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-alarm-gen: initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-snmp-collector: initializing (Database:RabbitMQ[] connection down)
contrail-topology: initializing (Database:RabbitMQ[] connection down)
2017-10-04 Wed 22:26:18:617.281 UTC server6 [Thread 140106626250496, Pid 3363]: server6:Global: MessageIndexTableInsert: Addition of message: AnalyticsApiStats, message UUID: f6301180-f3cb-4505-90b6-0ee26ebfb695 to table: MessageTableTimestamp FAILED
2017-10-04 Wed 22:26:18:618.250 UTC server6 [Thread 140106626250496, Pid 3363]: server6:Global: ObjectTableInsert: Addition of server4, message UUID f6301180-f3cb-4505-90b6-0ee26ebfb695 into table ObjectCollectorInfo FAILED
10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-analytics-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Redis-UVE name = 10.10.0.9:6381 server_addrs = [ 10.10.0.9:6381, ] status = Initializing >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Zookeeper name = OpServer server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10/04/2017 10:18:11 AM [contrail-snmp-collector]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state10.10.0.7:2181, ] status = Initializing description = >>, << type = Redis-UVE name = 10.10.0.7:6381 server_addrs = [ 10.10.0.7:6381, ] status = Initializing >>, << type = UvePartitions name = UVE-Aggregation server_addrs = [ ] status = Initializing >>, << type = Redis-UVE name = 10.10.0.8:6381 server_addrs = [ 10.10.0.8:6381, ] status = Initializing >>, ] description = Redis-UVE:10.10.0.9:6381[None], Collector, Zookeeper:OpServer[], Redis-UVE:10.10.0.7:6381[None], UvePartitions:UVE-Aggregation[None], Redis-UVE:10.10.0.8:6381[None] connection down >>, ] >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = server6:Analytics:contrail-analytics-api:0 client_info = << status = Idle successful_connections = 0 pid = 3329 http_port = 8090 start_time = 1507112275378607 collector_name = collector_ip = collector_list = [ 10.10.0.8:8086, 10.10.0.7:8086, 10.10.0.9:8086, ] >> sm_queue_count = 6 max_sm_queue_count = 6 >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: Exception: get_cql_session Failure ('Unable to connect to any servers', {'10.10.0.7': error(111, "Tried connecting to [('10.10.0.7', 9042)]. Last error: Connection refused"), '10.10.0.9': TypeError('ref() does not take keyword arguments',), '10.10.0.8': TypeError('ref() does not take keyword arguments',)})
10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.10.0.7:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.10.0.5:5672 10.10.0.6:5672 10.10.0.4:5672, ] status = Initializing description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = server6:Analytics:contrail-snmp-collector:0 client_info = << status = Idle successful_connections = 0 pid = 3445 http_port = 5920 start_time = 1507112283024981 collector_name = collector_ip = 10.10.0.7:8086 collector_list = [ 10.10.0.7:8086, 10.10.0.9:8086, 10.10.0.8:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvIdleHoldTimerExpired] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Session Event: TCP Connected
[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.10.0.9:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.10.0.5:5672 10.10.0.6:5672 10.10.0.4:5672, ] status = Initializing description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >>
Logs uploaded to /cs-shared/bugs/1721416.
The logs do not have enough information.
Is this reproducible?
Can I get access to a system in this state?