analytics_ip is incorrectly provisioned in contrail-svc-monitor.conf

[Vedamurthy Joshi]

R2.1 Build 6 ubuntu multi-node setup

In contrail_provisioning/config/common.py , the analytics_server ip is chosen as the internal ip.
It is possible that analytics need not run on the same node as config node.
Regression setup hit this problem and svc-monitor scheduling of netns instances failed

    def fixup_svc_monitor_config_file(self):
        # contrail-svc-monitor.conf
        template_vals = {'__contrail_ifmap_server_ip__': self.cfgm_ip,
                         '__contrail_ifmap_server_port__': '8444' if self._args.use_certs else '8443',
                         '__contrail_ifmap_username__': 'svc-monitor',
                         '__contrail_ifmap_password__': 'svc-monitor',
                         '__contrail_api_server_ip__': self._args.internal_vip or self.cfgm_ip,
                         '__contrail_api_server_port__': '8082',
                         '__contrail_analytics_server_ip__': self._args.internal_vip or self._args.collector_ip, <<<<<<<<<<<<<

env.roledefs = {
    'all': [host1, host2, host3, host4, host5, host6],
    'cfgm': [host1, host2],
    'openstack': [host2],
    'webui': [host3],
    'control': [host1, host3],
    'compute': [host4, host5, host6],
    'vgw': [host4, host5],
    'collector': [host1, host3],
    'database': [host1, host2, host3],
    'build': [host_build],
}

env.hostnames = {
    'all': ['nodea35', 'nodea34', 'nodec53', 'nodec54', 'nodec55', 'nodec56']
}

-----
svc-monitor.err:

  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 792, in process_poll_result
    func(idents)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 687, in _addmsg_service_instance_service_template
    self._create_svc_instance(st_obj, si_obj)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/svc_monitor.py", line 532, in _create_svc_instance
    self.netns_manager.create_service(st_obj, si_obj)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/instance_manager.py", line 516, in create_service
    si_obj.uuid, vm_obj.uuid)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/scheduler/vrouter_scheduler.py", line 131, in schedule
    candidates = self._get_candidates(si_uuid, vm_uuid)
  File "/usr/lib/python2.7/dist-packages/svc_monitor/scheduler/vrouter_scheduler.py", line 76, in _get_candidates
    if self.vrouter_running(vr['fq_name'][-1])]
  File "/usr/lib/python2.7/dist-packages/svc_monitor/scheduler/vrouter_scheduler.py", line 103, in vrouter_running
    vrouter_status = self._analytics.request(path, fqdn_uuid)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/analytics_client.py", line 45, in request
    resp = requests.get(url, **req_params)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 55, in get
    return request('get', url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/api.py", line 44, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 383, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/sessions.py", line 486, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/dist-packages/requests/adapters.py", line 378, in send
    raise ConnectionError(e)
ConnectionError: HTTPConnectionPool(host='10.204.216.30', port=8081): Max retries exceeded with url: /analytics/uves/vrouter/nodec54?cfilt=NodeStatus (Caused by <class 'socket.error'>: [Errno 111] Connection refused)

[Ignatious Johnson Christopher (ijohnson-x)]

Is this a HA setup? Also the requeat is sent to 10.204.216.30:8081 is that a VIP or physical IP of host1?

[Vedamurthy Joshi (vedujoshi)]

It is not configured as a "HA" setup...i.e no testbed.py config for HA

That is the physical IP of the host.

[Ignatious Johnson Christopher (ijohnson-x)]

As per the code (contrail-fabric-utils/contrail-provisioning), if it is not a HA setup, internal_vip will not be passed as argument to setup-vnc-config and so the collector_ip(collector node ip) only will be set in the contrail-svc-monitor.conf

As per the trace back logged in this bug, the connection attempt is made to "10.204.216.30" which is the first collector node.
So i see no issue with provisioning.

I am provisioning my multi node setup with latest 2.10 build , will run a lbass test and try to recreate the issue.

[Ignatious Johnson Christopher (ijohnson-x)]

Fix submitted for review https://review.opencontrail.org/#/c/6850/1/fabfile/tasks/provision.py

[Ignatious Johnson Christopher (ijohnson-x)]

Fixed with commit https://github.com/Juniper/contrail-fabric-utils/commit/e83c59edecfadeddf64aaafedb4fc13e21df3b2c in R2.1 branch.