API does not connect to IFMAP after ifmap-server is restarted

[Ankit Chadha]

API server doesn't reconnect to IFMAP after the IFMAP server is restarted. Contrail-status shows all services as 'active'.

Impacts 3.2 (seen on 3.2.0.0-19~kilo)

Workaround: Restart API server

[Ankit Chadha (achadha)]

Greenbelt traces:

  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/vnc_greenlets.py", line 32, in super_greenlet_handler
    greenlet_handler()
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 500, in _ifmap_dequeue_task
    self._publish_to_ifmap_dequeue()
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 562, in _publish_to_ifmap_dequeue
    _publish(requests, traces)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 513, in _publish
    ok, err_msg = self._publish_to_ifmap(''.join(requests))
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_ifmap.py", line 574, in _publish_to_ifmap
    resp_xml = self._mapclient.call('publish', req_xml)
  File "/usr/lib/python2.7/dist-packages/cfgm_common/ifmap/client.py", line 196, in call
    response = self._http.post('/', body = xml, headers = headers)
  File "/usr/lib/python2.7/dist-packages/geventhttpclient/client.py", line 185, in post
    return self.request(METHOD_POST, request_uri, body=body, headers=headers)
  File "/usr/lib/python2.7/dist-packages/geventhttpclient/client.py", line 152, in request
    sock = self._connection_pool.get_socket()
  File "/usr/lib/python2.7/dist-packages/geventhttpclient/connectionpool.py", line 114, in get_socket
    self._semaphore.acquire()
  File "/usr/lib/python2.7/dist-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)

[Édouard Thuleau (ethuleau)]

I tried to reproduce it on a dev/test platform which is running branch R3.2 and I'm not able to reproduce it.
Test:
- start VNC API server in debug level logging
- check ifmap graph populated on irond server (thanks to the ifmap-view command)
- restart the irond service only
- wait at least 60 seconds and see logs which say last ifmap health check failed, reconnect to irond then re-populate it
- check again ifmap graph populated on irond server (thanks to the ifmap-view command)

Perhaps you did not wait enough time the health check takes action. It runs every 60 seconds by default but you can tune it with contrail-api.conf's flag 'ifmap_health_check_interval' in the default section. Btw, if an action on graph appears before the health check, the ifmap graph should be repopulated without waiting health check.

Where did you get the greenlet trace above? From the VNC API introspect page?
Can I access your platform to check you have code which integrates that fix [1] & [2]?

[1] https://bugs.launchpad.net/opencontrail/+bug/1622101
[2] https://review.opencontrail.org/#/c/22802/

[Édouard Thuleau (ethuleau)]

The issue was due to a buggy version of geventhttpclient [1] python package used by the ifmap python client og the VNC api server. The version used here is 1.1.0. The bug was introduced in that 1.1.0 release and fixed since 1.3.0.
So we have to use version under 1.1.0 (1.0a0 tested and works) or at least upper that 1.3.0.

[1] https://github.com/gwik/geventhttpclient/pull/69

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28683
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28684
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28683
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/28684
Submitter: Ignatious Johnson Christopher (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28684
Committed: http://github.org/Juniper/contrail-packaging/commit/948155a76b9860b949b9c9071d5122cd785b878c
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 948155a76b9860b949b9c9071d5122cd785b878c
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Feb 10 17:14:50 2017 -0800

Updating python-geventhttpclient to 1.0a to

avoid https://github.com/gwik/geventhttpclient/pull/69

Change-Id: Ib987de7038d3c3585b51129ad03f0cb15e83e974
Closes-Bug: 1660454

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/28683
Committed: http://github.org/Juniper/contrail-packaging/commit/7443db6f12b6c0a1114a8638e0c637775642db4a
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7443db6f12b6c0a1114a8638e0c637775642db4a
Author: Ignatious Johnson Christopher <email address hidden>
Date: Fri Feb 10 17:14:50 2017 -0800

Updating python-geventhttpclient to 1.0a to

avoid https://github.com/gwik/geventhttpclient/pull/69

Change-Id: Ib987de7038d3c3585b51129ad03f0cb15e83e974
Closes-Bug: 1660454