SM:R3.0.2.x:Build 11: contrail-snmp-collector initializing (ApiServer:SNMP[Not connected] connection down)

[sundarkh]

SM:R3.0.2.x:Build 11: contrail-snmp-collector initializing (ApiServer:SNMP[Not connected] connection down)

1) Install SM : R3.0.2.x Build 11 ,
2)reimage the targets with ubuntu 14-04, and issue provision with kilo 3.0.2.x build 11
3) Issue Provision ; Observe that the provision gets completed successfully.
4) In the analytics node, snmp-collector is INITIALISING STATE (ApiServer:SNMP[Not connected]

SNMP_collector.log

11/11/2016 02:46:38 AM [contrail-snmp-collector]: Starting Introspect on HTTP Port 5920
11/11/2016 02:46:38 AM [contrail-snmp-collector]: Cannot write http_port 5920 to /tmp/contrail-snmp-collector.14575.http_port
11/11/2016 02:46:38 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodec33 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ , ] status = Down description = none to Idle on EvStart >>, << type = Discovery name = Collector server_addrs = [ 192.168.100.4:5998, ] status = Up description = Subscribe Response >>, ] description = Collector connection down >>, ] >>
11/11/2016 02:46:38 AM [contrail-snmp-collector]: SANDESH: [DROP: WrongClientSMState] NodeStatusUVE: data = << name = nodec33 process_status = [ << module_id = contrail-snmp-collector instance_id = 0 state = Non-Functional connection_infos = [ << type = Collector name = server_addrs = [ 192.168.100.3:8086, ] status = Initializing description = Idle to Connect on EvCollectorChange >>, << type = Discovery name = Collector server_addrs = [ 192.168.100.4:5998, ] status = Up description = Subscribe Response >>, ] description = Collector connection down >>, ] >>

Setup

Config Nodes : [u'nodec35', u'nodec33']
Control Nodes : [u'nodec35', u'nodec33']
Compute Nodes : [u'nodea4', u'nodec57']
Openstack Node : nodec35
WebUI Node : nodec33
Analytics Nodes : [u'nodec33']

Notes

-----

Issue not seen in Single node setup

Sanity Test case AnalyticsTestSanity.test_contrail_status, fails because of this issue

[Nitish Krishna Kaveri (nitishk)]

I had made a change for this issue before to this branch:
https://review.opencontrail.org/#/c/23750/
This change was reviewed but not approved and so it got auto-abandoned.
When check-ins do not get approved in time it is an issue.

[Nitish Krishna Kaveri (nitishk)]

I am going to make two check-ins that are needed related to this scenario (separate collector and config)

https://bugs.launchpad.net/juniperopenstack/+bug/1616666
https://bugs.launchpad.net/juniperopenstack/+bug/1616665
https://bugs.launchpad.net/juniperopenstack/+bug/1618574

All three bugs listed above were opened when dev ops folks tested separate config and collector roles on different nodes. These were debugged and fixed in 3.0 and 3.1 (mainline at the time)

The reason this bug is seen now is not because of any new code change but because this scenario of separate analytics and config was not tested before on 3.0.2.x

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/26013
Submitter: Nitish Krishna Kaveri (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/26013
Committed: http://github.org/Juniper/contrail-puppet/commit/639df223d963040a5b887eb6ed7b21d1a1fd153b
Submitter: Zuul
Branch: R3.0.2.x

commit 639df223d963040a5b887eb6ed7b21d1a1fd153b
Author: nitishkrishna <email address hidden>
Date: Tue Aug 30 12:16:30 2016 -0700

Closes-Bug: #1616665 - Missing keystone file, api related params in separate collector scenario

Closes-Bug: #1616666

Three issues:
1. contrail-keystone-auth.conf missing on separate collector/db node
2. Wrong vip set in contrail-snmp-collector.conf if contrail_internal_vip set
3. Missing api_server param in contrail-analytics-api.conf

Closes-Bug: #1641093 - Previously fixed bug now to be fixed in 3.0.2.x

I have cherry picked:

1. https://review.opencontrail.org/#/c/23777/
2. https://review.opencontrail.org/#/c/23746/

Into R3.0.2.x

The bugs above are now fixed in R3.0.2.x also

Please note:
Bug 1618574 will not be fixed in R3.0.2.x (discussed with Megh)

Change-Id: I28fdce93422bf62d6c7f07c81a3fda7874c2d931