SM provisioning 3 node HA provisioning Contrail-API server stay down

[Sarath]

Version: 4.0.1.0-46-mitaka
Topology: 3 node HA (contrail/Openstack) with multiple Esxi & Kvm

Provisioning completed but found API servers down on all 3 nodes.

Development Dheeraj triaged the setup and found the "internal VIP" also mandatory to be provided in JSON
as some components dependency on that.

Note: After configuring both internal_vip and contrail_internal_vip which are just same as their external
this issue got resolved.

Please find more debugging details on this,

root@5a10s31:~#
root@5a10s31:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c6e500e96ab2 10.87.36.15:5100/contrail_4_0_1_50_vc_new-contrail-analytics:4.0.1-50 "/bin/sh -c /entry..." 5 hours ago Up 5 hours analytics
ff1c4eeec510 10.87.36.15:5100/contrail_4_0_1_50_vc_new-contrail-analyticsdb:4.0.1-50 "/bin/sh -c /entry..." 5 hours ago Up 5 hours analyticsdb
6cfa8ee44d12 10.87.36.15:5100/contrail_4_0_1_50_vc_new-contrail-controller:4.0.1-50 "/bin/sh -c /entry..." 5 hours ago Up 5 hours controller
root@5a10s31:~# openstack-status | grep active | wc -l
docker exec -it controller bash
Warning keystonerc not sourced
19
root@5a10s31:~# docker exec -it controller bash
contrail-status
root@5a10s31(controller):/# contrail-status
exit
== Contrail Control ==
contrail-control active
contrail-control-nodemgr initializing (NTP state unsynchronized.)
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 initializing (Generic Connection:Keystone[] connection down)
contrail-config-nodemgr initializing (NTP state unsynchronized.)
contrail-device-manager active
contrail-schema active
contrail-svc-monitor initializing

== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

root@5a10s31(controller):/# exit
exit
root@5a10s31:~# docker exec -it analytics bash
contrail-status
root@5a10s31(analytics):/# contrail-status
exit
docker exec -it analyticsdb bash
== Contrail Analytics ==
contrail-alarm-gen initializing (ApiServer:Config[Config sync failed] connection down)
contrail-analytics-api active
contrail-analytics-nodemgr initializing (NTP state unsynchronized.)
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

root@5a10s31(analytics):/# exit
exit

From: Sarathbabu Narasimhan
Sent: Wednesday, August 30, 2017 6:14 PM
To: Dheeraj Gautam <email address hidden>
Cc: Abhay Joshi <email address hidden>; Jeba Paulaiyan <email address hidden>; Sudheendra Rao <email address hidden>; Sandip Dey <email address hidden>; Kamlesh Parmar <email address hidden>; Sarathbabu Narasimhan <email address hidden>
Subject: RE: Meeting to discuss Ocata and Vcenter

Thanks Dheeraj for triaging the issue and sure I just added both internal_vip and contrail_internet_vip to be same as provided external values and
re-issued provisioning.

If this JSON changes gets this success, then we can ask Nitish to update bug# 1713592 for documentation.

Thanks
*Sarath

From: Dheeraj Gautam
Sent: Wednesday, August 30, 2017 5:49 PM
To: Sarathbabu Narasimhan <email address hidden>
Cc: Abhay Joshi <email address hidden>; Jeba Paulaiyan <email address hidden>; Sudheendra Rao <email address hidden>; Sandip Dey <email address hidden>; Kamlesh Parmar <email address hidden>
Subject: Re: Meeting to discuss Ocata and Vcenter

Looked at the setup.

This issue is due to specifying external_vip only and leaving internal_vip empty.

Contrail is correctly to use vip configured for keystone, but older puppet code give precedence to internal_vip and configured endpoints to openstack[0] and haproxy is not configured

@Sarath,
We need to specify internal_vip as well (under openstack::ha::internal_vip).

I am not sure if we need to specify contrail_internal_vip as well. It would be good to specify this as well.

-Dheeraj

From: Sarathbabu Narasimhan <email address hidden>
Date: Wednesday, August 30, 2017 at 4:29 PM
To: "<email address hidden>" <email address hidden>
Cc: Abhay Joshi <email address hidden>, Jeba Paulaiyan <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>, Kamlesh Parmar <email address hidden>, Sarathbabu Narasimhan <email address hidden>
Subject: RE: Meeting to discuss Ocata and Vcenter

Hi Dheeraj,

In Vcenter-as-compute, with latest build#50 and with #1713592 patches applied, provisioning shown completed but we have API server down issue due to keystone issue
and looks provisioning not picking the password per JSON files, Please find below,

TargetNode : 10.87.36.10 ( sm ip 10.87.36.15 )
#########

[Abhay Joshi (abhayj)]

The bug is opened for documentation purpose.

[Abhay Joshi (abhayj)]

From: Sarathbabu Narasimhan <email address hidden>
Date: Thursday, August 31, 2017 at 2:54 PM
To: Dheeraj Gautam <email address hidden>
Cc: Abhay Joshi <email address hidden>, Jeba Paulaiyan <email address hidden>, Sudheendra Rao <email address hidden>, Sandip Dey <email address hidden>, Kamlesh Parmar <email address hidden>
Subject: RE: Meeting to discuss Ocata and Vcenter

Thanks Dheeraj and now Vcenter-as-compute provisioning success  with latest build#50(Mitaka) and all the services good
and also just started automated Sanity Functionality verification testcases and seeing they are too passing and progressing.

For tracking this new issue which “internal_vip being mandatory”,  please find the bug below,
Bug #1714364 : SM provisioning 3 node HA provisioning Contrail-API server stay down

Thanks
*Sarath

From: Sarathbabu Narasimhan
Sent: Wednesday, August 30, 2017 6:14 PM
To: Dheeraj Gautam <email address hidden>
Cc: Abhay Joshi <email address hidden>; Jeba Paulaiyan <email address hidden>; Sudheendra Rao <email address hidden>; Sandip Dey <email address hidden>; Kamlesh Parmar <email address hidden>; Sarathbabu Narasimhan <email address hidden>
Subject: RE: Meeting to discuss Ocata and Vcenter

Thanks Dheeraj for triaging the issue and sure I just added both internal_vip and contrail_internet_vip to be same as provided external values and
re-issued provisioning.

If this JSON changes gets this success, then we can ask Nitish to update bug# 1713592 for documentation.

Thanks
*Sarath

[Sarath (nsarath)]

As given code/documentation results in critical solution failure, the fix may either change of code (or) change of documentation and development decided to fix provided through update of documentation,

Please find below RN details,

"RN: When SM provisioning for Contrail/Openstack HA Topology with Single interface (or) multiple interface as below must be provided part of JSON file inputs,

Single interface:-

If more than one openstack node:
Both internal and external vips are mandatory.
They must be the same.

Multiple interface:-

If more than one openstack node:
Both internal and external vips are mandatory.
They cannot be the same.
"