config-openstack: Need to ensure connection to api-server in api-hook

[Hampapur Ajay]

Following sequence of events happened causing issue in post-virtual-network-* actions in ResourceApiDriver of vnc_openstack

1. On a newly provisioned system, api-server is launched
2. Due to race condition with keystone start/provisioning, the api-server connection from ResourceApiDriver::_get_vnc_conn() has authentication issue and the greenlet dies
3. Later on a network is created via neutron API
4. the post_virtual_network_create fails since there is no connection to api-server i.e. vnc_lib is None
5. same issue exists for project syncing

ERROR:server13:contrail-api:Config:0:Connection to API server failed: Unable to connect to keystone for authentication. Verify keystone server details
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 728, in _get_api_connection
    tenant_name=self._admin_tenant)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 302, in __init__
    retry_on_error=False)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 665, in _request
    self._headers = self._authenticate(content, self._headers)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 485, in _authenticate
    raise RuntimeError('Unable to connect to keystone for authentication. Verify keystone server details')
RuntimeError: Unable to connect to keystone for authentication. Verify keystone server details
<Greenlet at 0x7f529d6c9910: <bound method ResourceApiDriver._get_api_connection of <vnc_openstack.ResourceApiDriver object at 0x7f529d5a25d0>>> failed with RuntimeError

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 452, in http_resource_create
    'post_%s_create' %(obj_type), obj_dict)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 244, in map_method
    method_name, *args, **kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 217, in map
    self._invoke_one_plugin(response.append, func, e, args, kwds)
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 248, in _invoke_one_plugin
    response_callback(func(e, *args, **kwds))
  File "/usr/lib/python2.7/dist-packages/stevedore/extension.py", line 222, in _call_extension_method
    return getattr(extension.obj, method_name)(*args, **kwds)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/__init__.py", line 813, in post_virtual_network_create
    self._vnc_lib.kv_store(subnet_uuid, subnet_name)
AttributeError: 'NoneType' object has no attribute 'kv_store'

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14668
Submitter: Hampapur Ajay (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14668
Committed: http://github.org/Juniper/contrail-controller/commit/a3e2adb241fd4d55768492aa33a598f6f69e90d5
Submitter: Zuul
Branch: master

commit a3e2adb241fd4d55768492aa33a598f6f69e90d5
Author: Hampapur Ajay <email address hidden>
Date: Tue Oct 27 17:13:47 2015 -0700

Retry connection to api-server on exceptions; have hooks wait for connection

ResourceApiDriver needs to retry connection to api-server on any
exception (like authentication error etc.) and any pre/post hook
handlers should wait till connection is established to execute any
updates.

Change-Id: Iddc0ff6b2898bb0b02f4fd36b6e2ce746e8bc0e0
Closes-Bug: #1508558

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/14713
Submitter: Hampapur Ajay (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/14713
Committed: http://github.org/Juniper/contrail-controller/commit/3da6e824fbbac2fd15d5728629e443df1fc7f41a
Submitter: Zuul
Branch: R2.20

commit 3da6e824fbbac2fd15d5728629e443df1fc7f41a
Author: Hampapur Ajay <email address hidden>
Date: Tue Oct 27 17:13:47 2015 -0700

Retry connection to api-server on exceptions; have hooks wait for connection

ResourceApiDriver needs to retry connection to api-server on any
exception (like authentication error etc.) and any pre/post hook
handlers should wait till connection is established to execute any
updates.

Change-Id: Iddc0ff6b2898bb0b02f4fd36b6e2ce746e8bc0e0
Closes-Bug: #1508558
(cherry picked from commit a3e2adb241fd4d55768492aa33a598f6f69e90d5)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/18578
Submitter: Antoine Eiche (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/18578
Committed: http://github.org/Juniper/contrail-controller/commit/7295d67e159fd32f0abab9230559f6709ff13894
Submitter: Zuul
Branch: R2.21.x

commit 7295d67e159fd32f0abab9230559f6709ff13894
Author: Hampapur Ajay <email address hidden>
Date: Tue Oct 27 17:13:47 2015 -0700

Retry connection to api-server on exceptions; have hooks wait for connection

ResourceApiDriver needs to retry connection to api-server on any
exception (like authentication error etc.) and any pre/post hook
handlers should wait till connection is established to execute any
updates.

Conflicts:
 src/config/vnc_openstack/vnc_openstack/__init__.py
(cherry picked from commit a3e2adb241fd4d55768492aa33a598f6f69e90d5)

Change-Id: Iddc0ff6b2898bb0b02f4fd36b6e2ce746e8bc0e0
Closes-Bug: #1508558