Juniper Openstack

Comment 0 for bug 1721416

Revision history for this message

wenqing liang (wliang) wrote on 2017-10-04:

CB r4.1-8 newton:

== Contrail Analytics ==
contrail-collector: initializing (Database:server6:Global connection down)
contrail-analytics-api: initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-alarm-gen: initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-snmp-collector: initializing (Database:RabbitMQ[] connection down)
contrail-topology: initializing (Database:RabbitMQ[] connection down)

2017-10-04 Wed 22:26:18:617.281 UTC server6 [Thread 140106626250496, Pid 3363]: server6:Global: MessageIndexTableInsert: Addition of message: AnalyticsApiStats, message UUID: f6301180-f3cb-4505-90b6-0ee26ebfb695 to table: MessageTableTimestamp FAILED
2017-10-04 Wed 22:26:18:618.250 UTC server6 [Thread 140106626250496, Pid 3363]: server6:Global: ObjectTableInsert: Addition of server4, message UUID f6301180-f3cb-4505-90b6-0ee26ebfb695 into table ObjectCollectorInfo FAILED

10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-analytics-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Redis-UVE name = 10.10.0.9:6381 server_addrs = [ 10.10.0.9:6381, ] status = Initializing >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Zookeeper name = OpServer server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10/04/2017 10:18:11 AM [contrail-snmp-collector]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state10.10.0.7:2181, ] status = Initializing description = >>, << type = Redis-UVE name = 10.10.0.7:6381 server_addrs = [ 10.10.0.7:6381, ] status = Initializing >>, << type = UvePartitions name = UVE-Aggregation server_addrs = [ ] status = Initializing >>, << type = Redis-UVE name = 10.10.0.8:6381 server_addrs = [ 10.10.0.8:6381, ] status = Initializing >>, ] description = Redis-UVE:10.10.0.9:6381[None], Collector, Zookeeper:OpServer[], Redis-UVE:10.10.0.7:6381[None], UvePartitions:UVE-Aggregation[None], Redis-UVE:10.10.0.8:6381[None] connection down >>, ] >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = server6:Analytics:contrail-analytics-api:0 client_info = << status = Idle successful_connections = 0 pid = 3329 http_port = 8090 start_time = 1507112275378607 collector_name = collector_ip = collector_list = [ 10.10.0.8:8086, 10.10.0.7:8086, 10.10.0.9:8086, ] >> sm_queue_count = 6 max_sm_queue_count = 6 >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: Exception: get_cql_session Failure ('Unable to connect to any servers', {'10.10.0.7': error(111, "Tried connecting to [('10.10.0.7', 9042)]. Last error: Connection refused"), '10.10.0.9': TypeError('ref() does not take keyword arguments',), '10.10.0.8': TypeError('ref() does not take keyword arguments',)})

10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.10.0.7:8086, ] status = Down description = Connect to Idle on EvTcpConnectFail >>, << type = Database name = RabbitMQ server_addrs = [ 10.10.0.5:5672 10.10.0.6:5672 10.10.0.4:5672, ] status = Initializing description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = server6:Analytics:contrail-snmp-collector:0 client_info = << status = Idle successful_connections = 0 pid = 3445 http_port = 5920 start_time = 1507112283024981 collector_name = collector_ip = 10.10.0.7:8086 collector_list = [ 10.10.0.7:8086, 10.10.0.9:8086, 10.10.0.8:8086, ] >> sm_queue_count = 1 max_sm_queue_count = 3 >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvIdleHoldTimerExpired] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Session Event: TCP Connected
[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = server6 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ 10.10.0.9:8086, ] status = Initializing description = Idle to Connect on EvIdleHoldTimerExpired >>, << type = Database name = RabbitMQ server_addrs = [ 10.10.0.5:5672 10.10.0.6:5672 10.10.0.4:5672, ] status = Initializing description = >>, ] description = Collector, Database:RabbitMQ[] connection down >>, ] >>

CB r4.1-8 newton:

== Contrail Analytics ==
contrail-collector:           initializing (Database:server6:Global connection down)
contrail-analytics-api:       initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-alarm-gen:           initializing (Redis-UVE:10.10.0.7:6381[None], Redis-UVE:10.10.0.8:6381[None] connection down)
contrail-snmp-collector:      initializing (Database:RabbitMQ[] connection down)
contrail-topology:            initializing (Database:RabbitMQ[] connection down)

2017-10-04 Wed 22:26:18:617.281 UTC  server6 [Thread 140106626250496, Pid 3363]: server6:Global: MessageIndexTableInsert: Addition of message: AnalyticsApiStats, message UUID: f6301180-f3cb-4505-90b6-0ee26ebfb695 to table: MessageTableTimestamp FAILED
2017-10-04 Wed 22:26:18:618.250 UTC  server6 [Thread 140106626250496, Pid 3363]: server6:Global: ObjectTableInsert: Addition of server4, message UUID f6301180-f3cb-4505-90b6-0ee26ebfb695 into table ObjectCollectorInfo FAILED

10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = server6  process_status = [ <<  module_id = contrail-analytics-api  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Redis-UVE  name = 10.10.0.9:6381  server_addrs = [ 10.10.0.9:6381,  ]  status = Initializing  >>, <<  type = Collector  name =   server_addrs = [ ,  ]  status = Down  description = none to Idle on EvStart  >>, <<  type = Zookeeper  name = OpServer  server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10/04/2017 10:18:11 AM [contrail-snmp-collector]: Sandesh Client: Event[EvIdleHoldTimerExpired] => State[Idle] -> State[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state10.10.0.7:2181,  ]  status = Initializing  description =   >>, <<  type = Redis-UVE  name = 10.10.0.7:6381  server_addrs = [ 10.10.0.7:6381,  ]  status = Initializing  >>, <<  type = UvePartitions  name = UVE-Aggregation  server_addrs = [  ]  status = Initializing  >>, <<  type = Redis-UVE  name = 10.10.0.8:6381  server_addrs = [ 10.10.0.8:6381,  ]  status = Initializing  >>,  ]  description = Redis-UVE:10.10.0.9:6381[None], Collector, Zookeeper:OpServer[], Redis-UVE:10.10.0.7:6381[None], UvePartitions:UVE-Aggregation[None], Redis-UVE:10.10.0.8:6381[None] connection down  >>,  ]  >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = server6:Analytics:contrail-analytics-api:0  client_info = <<  status = Idle  successful_connections = 0  pid = 3329  http_port = 8090  start_time = 1507112275378607  collector_name =   collector_ip =   collector_list = [ 10.10.0.8:8086, 10.10.0.7:8086, 10.10.0.9:8086,  ]  >>  sm_queue_count = 6  max_sm_queue_count = 6  >>
10/04/2017 10:17:55 AM [contrail-analytics-api]: Exception: get_cql_session Failure ('Unable to connect to any servers', {'10.10.0.7': error(111, "Tried connecting to [('10.10.0.7', 9042)]. Last error: Connection refused"), '10.10.0.9': TypeError('ref() does not take keyword arguments',), '10.10.0.8': TypeError('ref() does not take keyword arguments',)})

10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = server6  process_status = [ <<  module_id = contrail-snmp-collector  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181,  ]  status = Up  description =   >>, <<  type = Collector  name =   server_addrs = [ 10.10.0.7:8086,  ]  status = Down  description = Connect to Idle on EvTcpConnectFail  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.10.0.5:5672  10.10.0.6:5672  10.10.0.4:5672,  ]  status = Initializing  description =   >>,  ]  description = Collector, Database:RabbitMQ[] connection down  >>,  ]  >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Processing event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:07 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = <<  name = server6:Analytics:contrail-snmp-collector:0  client_info = <<  status = Idle  successful_connections = 0  pid = 3445  http_port = 5920  start_time = 1507112283024981  collector_name =   collector_ip = 10.10.0.7:8086  collector_list = [ 10.10.0.7:8086, 10.10.0.9:8086, 10.10.0.8:8086,  ]  >>  sm_queue_count = 1  max_sm_queue_count = 3  >>
10/04/2017 10:18:07 AM [contrail-snmp-collector]: Discarding event[EvSandeshUVESend] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Processing event[EvIdleHoldTimerExpired] in state[Idle]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: Session Event: TCP Connected
[Connect]
10/04/2017 10:18:11 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = <<  name = server6  process_status = [ <<  module_id = contrail-snmp-collector  instance_id = 0  state = Non-Functional  connection_infos = [ <<  type = Zookeeper  name = Zookeeper  server_addrs = [ 10.10.0.8:2181, 10.10.0.9:2181, 10.10.0.7:2181,  ]  status = Up  description =   >>, <<  type = Collector  name =   server_addrs = [ 10.10.0.9:8086,  ]  status = Initializing  description = Idle to Connect on EvIdleHoldTimerExpired  >>, <<  type = Database  name = RabbitMQ  server_addrs = [ 10.10.0.5:5672  10.10.0.6:5672  10.10.0.4:5672,  ]  status = Initializing  description =   >>,  ]  description = Collector, Database:RabbitMQ[] connection down  >>,  ]  >>