Vcenter-as-compute: contrail-api initializing on all HA nodes due to keystone issue after provisioning latest mainline #84

[Sarath]

Version : Mainline #84

This seen after provisioning complete and below logs/errors,

01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = 5a10s31 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.87.36.10:2181, 10.87.36.11:2181, 10.87.36.12:2181, ] status = Initializing description = >>, ] description = Zookeeper:Zookeeper[] connection down >>, ] >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = 5a10s31 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.87.36.10:2181, 10.87.36.11:2181, 10.87.36.12:2181, ] status = Initializing description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Zookeeper:Zookeeper[], Collector connection down >>, ] >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] SandeshModuleClientTrace: data = << name = 5a10s31:Config:contrail-api:0 client_info = << status = Idle successful_connections = 0 pid = 3874 http_port = 8084 start_time = 1515446572758927 collector_name = collector_ip = collector_list = [ 10.87.36.10:8086, 10.87.36.12:8086, 10.87.36.11:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 2 >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = 5a10s31 process_status = [ << module_id = contrail-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Zookeeper name = Zookeeper server_addrs = [ 10.87.36.10:2181, 10.87.36.11:2181, 10.87.36.12:2181, ] status = Up description = >>, << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, ] description = Collector connection down >>, ] >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 7165 response_size = 0 identifier = req-864bd319-b8f1-4a77-9368-69a993402b2e >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = GET_CHILDREN application = ZK response_time_in_usec = 12936 response_size = 0 identifier = req-864bd319-b8f1-4a77-9368-69a993402b2e >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = READ application = ZK response_time_in_usec = 8590 response_size = 0 identifier = req-864bd319-b8f1-4a77-9368-69a993402b2e >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = DELETE application = ZK response_time_in_usec = 21777 response_size = 0 identifier = req-864bd319-b8f1-4a77-9368-69a993402b2e >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_INFO]: VncApiLatencyStatsLog: node_name = issu-vm6 api_latency_stats = << operation_type = CREATE application = ZK response_time_in_usec = 98563 response_size = 0 identifier = req-864bd319-b8f1-4a77-9368-69a993402b2e >>
01/08/2018 01:22:52 PM [contrail-api]: SANDESH: [DROP: NoSession] __default__ [SYS_IN

root@5a10s30(analytics):/# contrail-status
exit
== Contrail Analytics ==
contrail-collector: active
contrail-analytics-api: active
contrail-query-engine: active
contrail-alarm-gen: active
contrail-snmp-collector: active
contrail-topology: active
contrail-analytics-nodemgr: active
root@5a10s30(analytics):/# exit
exit
root@5a10s30:~# ^H
-bash: :s^H: substitution failed
root@5a10s30:~# docker exec -it analyticsdb bash
contrail-status
contrail-status
root@5a10s30(analyticsdb):/# contrail-status
exit
== Contrail Database ==
contrail-database: active

kafka: active
contrail-database-nodemgr: active
root@5a10s30(analyticsdb):/# exit
exit
root@5a10s30:~# docker exec -it controller bash
contrail-status
root@5a10s30(controller):/# contrail-status
exit
== Contrail Control ==
contrail-control: active
contrail-named: active
contrail-dns: active
contrail-control-nodemgr: active
== Contrail Config ==
contrail-api: initializing (Generic Connection:Keystone[] connection down)
contrail-schema: backup
contrail-svc-monitor: backup
contrail-device-manager: active
contrail-config-nodemgr: active
== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui: active
contrail-webui-middleware: active
== Contrail Support Services ==
zookeeper: active
rabbitmq-server: inactive (disabled on boot)
root@5a10s30(controller):/# exit
exit

[Sarath (nsarath)]

Workaround: To restart contrail-api service

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/38778
Submitter: Sachin Bansal (<email address hidden>)

[Sarath (nsarath)]

not consistently reproducible. removing sanityblocker and monitor for few builds