vcenter-as-compute: contrail-analytics-api service initializing seen one of HA node

[Sarath]

This seen on Vcenter-as-compute sanity with latest R4.0 #20 ( newton )
This issue seen soon after Fresh install provisioning.
This is 3 node HA setup and couple of nodes is okay and this issue seen only on one node

root@5a10s29:~# contrail-status
ssh root@10.87.36.21
== Contrail Control ==
contrail-control: active
contrail-named: active
contrail-dns: active
contrail-control-nodemgr: active
== Contrail Analytics ==
contrail-collector: active
contrail-analytics-api: initializing (UvePartitions:UVE-Aggregation[Partitions:0] connection down)
contrail-query-engine: active
contrail-alarm-gen: active
contrail-snmp-collector: active
contrail-topology: active
contrail-analytics-nodemgr: active
== Contrail Config ==
contrail-api: active
contrail-schema: active
contrail-svc-monitor: active
contrail-device-manager: backup
contrail-config-nodemgr: active
== Contrail Web UI ==
contrail-webui: active (disabled on boot)
contrail-webui-middleware: active (disabled on boot)
== Contrail Database ==
contrail-database: active

kafka: active
contrail-database-nodemgr: active
== Contrail Support Services ==
rabbitmq-server: active
Warning: zookeeper.service changed on disk. Run 'systemctl daemon-reload' to reload units.
zookeeper: active

>>>> logs

UVE-Aggregation server_addrs = [ ] status = Down description = Partitions:0 >>, ] description = UvePartitions:UVE-Aggregation[Partitions:0] connection down >>, ] >>
06/06/2017 03:31:14 PM [contrail-analytics-api]: Token info for null NOT FOUND
06/06/2017 03:31:14 PM [contrail-analytics-api]: Token info for null NOT FOUND
06/06/2017 03:31:18 PM [contrail-analytics-api]: Token info for null NOT FOUND
06/06/2017 04:26:10 PM [contrail-analytics-api]: Token info for 4e88c72ee40c40a7a8b28bef6ca4ee79 NOT FOUND
06/06/2017 04:26:11 PM [contrail-analytics-api]: Token info for 2f2e90da7cb24a4f969376dfd4fb4699 NOT FOUND
06/06/2017 04:26:11 PM [contrail-analytics-api]: Token info for 2f2e90da7cb24a4f969376dfd4fb4699 NOT FOUND
06/06/2017 04:26:15 PM [contrail-analytics-api]: Token info for 0f7f16f918c540aeb19d985583923f19 NOT FOUND
06/06/2017 04:26:15 PM [contrail-analytics-api]: Token info for 4157b506312e42b6987e8b38bcda9faf NOT FOUND
06/06/2017 04:26:18 PM [contrail-analytics-api]: Token info for 846afefe78d1433484948307f85adbaa NOT FOUND
06/06/2017 04:26:18 PM [contrail-analytics-api]: Token info for 846afefe78d1433484948307f85adbaa NOT FOUND
06/06/2017 04:26:18 PM [contrail-analytics-api]: Token info for 846afefe78d1433484948307f85adbaa NOT FOUND
06/06/2017 04:26:19 PM [contrail-analytics-api]: Token info for 765a764451f24b51ba9503a4b7bc19f3 NOT FOUND
06/06/2017 04:30:49 PM [contrail-analytics-api]: NodeStatusUVE: data = << name = 5a10s29 process_status = [ << module_id = contrail-analytics-api instance_id = 0 state = Non-Functional connection_infos = [ << type = Redis-UVE name = 10.87.36.12:6379 server_addrs = [ 10.87.36.12:6379, ] status = Up >>, << type = Collector name = server_addrs = [ 10.87.36.10:8086, ] status = Up description = ClientInit to Established on EvSandeshCtrlMessageRecv >>, << type = Zookeeper name = OpServer server_addrs = [ 10.87.36.10:2181, 10.87.36.11:2181, 10.87.36.12:2181, ] status = Up description = >>, << type = ApiServer name = server_addrs = [ 10.87.36.25:8082, ] status = Up description = >>, << type = Redis-UVE name = 10.87.36.11:6379 server_addrs = [ 10.87.36.11:6379, ] status = Up >>, << type = Redis-UVE name = 10.87.36.10:6379 server_addrs = [ 10.87.36.10:6379, ] status = Up >>, << type = UvePartitions name = UVE-Aggregation server_addrs = [ ] status = Down description = Partitions:0 >>, ] description = UvePartitions:UVE-Aggregation[Partitions:0] connection down >>, ] >>
06/06/2017 04:30:49 PM [contrail-analytics-api]: SandeshModuleClientTrace: data = << name = 5a10s29:Analytics:contrail-analytics-api:0 client_info = << status = Established successful_connections = 1 pid = 28877 http_port = 8090 start_time = 1496787966286132 collector_name = 5a10s31 collector_ip = 10.87.36.10:8086 collector_list = [ 10.87.36.10:8086, 10.87.36.11:8086, 10.87.36.12:8086, ] >> sm_queue_count = 2 max_sm_queue_count = 7 >>
06/06/2017 04:31:14 PM [contrail-analytics-api]: Token info for 0f7f16f918c540aeb19d985583923f19 NOT FOUND
06/06/2017 04:31:52 PM [contrail-analytics-api]: Token info for 70789104f8a94c14829c38cc3d6f0bd9 NOT FOUND
0

[Sarath (nsarath)]

nsarath@ubuntu-build04:/auto/cores/1696280$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 326215680 Jun 6 17:38 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 10352640 Jun 6 17:38 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 333762560 Jun 6 17:38 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 324157440 Jun 6 17:38 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 26132480 Jun 6 17:38 Esxi-1-log.tar
-rwxrwxrwx 1 nsarath test 26511360 Jun 6 17:38 Esxi-2-log.tar
-rwxrwxrwx 1 nsarath test 26511360 Jun 6 17:38 Esxi-3-log.tar
-rwxrwxrwx 1 nsarath test 17090560 Jun 6 17:38 Kvm-1-log.tar
-rwxrwxrwx 1 nsarath test 16015360 Jun 6 17:38 Nova-node-log.tar

[Anish Mehta (amehta00)]

Is this reproducible?

[Sarath (nsarath)]

Anish, do you want to try on R4.0 image?
If you need for R4.0.1, still dev-work-on-progress for sm provisioning and will verify once ready.

[Anish Mehta (amehta00)]

Sarath, are you able to try this again?

[Sarath (nsarath)]

Anish, R4.0.1 image still not available for vcenter-as-compute and will try once ready.

[Anish Mehta (amehta00)]

We are not looking at non-containerized install issue.
Please retest with containerized install, and open the bug if needed.