build-18-ubuntu-liberty: analytics-api didn't recover after reboot

[shajuvk]

Analytics api was stuck in listening state after rebooting all the nodes.

istening on http://0.0.0.0:9081/
Hit Ctrl-C to quit.

Bottle v0.12.0 server starting up (using GeventPoolServer(spawn=<Pool at 0x7f71d28a6390 set([])>))...
Listening on http://localhost:8181/
Hit Ctrl-C to quit.

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/gevent/greenlet.py", line 327, in run
    result = self._run(*self.args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/opserver/vnc_cfg_api_client.py", line 51, in connect
    auth_protocol=self._conf_info['auth_protocol'])
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 322, in __init__
    retry_on_error=False)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 741, in _request
    self._headers = self._authenticate(content, self._headers)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 575, in _authenticate
    raise RuntimeError('Authentication Failure')
RuntimeError: Authentication Failure
<Greenlet at 0x7f71d2d519b0: <bound method VncCfgApiClient.connect of <opserver.vnc_cfg_api_client.VncCfgApiClient object at 0x7f71d3da9e10>>> failed with RuntimeError

[shajuvk (shajuvk)]

root@a6s33:~# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-alarm-gen active
contrail-analytics-api initializing (ApiServer connection down)
contrail-analytics-nodemgr active
contrail-collector active
contrail-query-engine active
contrail-snmp-collector active
contrail-topology active

== Contrail Config ==

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23074
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23074
Committed: http://github.org/Juniper/contrail-controller/commit/8072aa5ffd37e4082d7ae9697020a6160e8d2682
Submitter: Zuul
Branch: R3.1

commit 8072aa5ffd37e4082d7ae9697020a6160e8d2682
Author: Megh Bhatt <email address hidden>
Date: Mon Aug 8 17:44:52 2016 -0700

Keep on trying to create VNC API client from analytics API

The gevent that creates the VNC API client was exiting due to
authentication failure exception. Changed code to handle all
exceptions and keep on trying to create the API client. The
node status will show the API connection down in case we are
not able to create the VNC API client.

Change-Id: I518f6ea6c660fd28ef1b66011554ef5ca5281c1a
Closes-Bug: #1611158

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22536
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/21917
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22536
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/21917
Committed: http://github.org/Juniper/contrail-controller/commit/051a2c1851419b7081db5e611a40ebc57235b52f
Submitter: Zuul
Branch: master

commit 051a2c1851419b7081db5e611a40ebc57235b52f
Author: Megh Bhatt <email address hidden>
Date: Tue Jul 12 16:34:17 2016 -0700

1. Rename multi_tenancy to aaa_mode for analytics API

Handle keystone v2 and v3 token infos returned by
VNC API. Enable cloud-admin-only aaa_mode by default

Change analytics DB and underlay to overlay mapper to
use local admin port when quering opserver

Do not cache auth_token in vnc lib

Closes-Bug: #1599654

2. Changes to bring analytics authenticated access in sync with config

1. Rename aaa_mode value cloud-admin-only to cloud-admin
2. CLOUD_ADMIN_ROLE defaults to admin instead of cloud-admin

Partial-Bug: #1607563
(cherry picked from commit 42db6e38e55bc2410297a99c2af3bea03faa938c)

3. Fix missing import of OpServerUtils in analytics_db.py

Closes-Bug: #1609054
(cherry picked from commit cf5f0567c9bb03e83cd83515b775d2018e668d0c)

4. Remove aaa_mode value cloud-admin-only

Closes-Bug: #1609987
(cherry picked from commit 58a8a0fe3a404b5e6a11b01008064b96ed66109e)

5. Keep on trying to create VNC API client from analytics API

The gevent that creates the VNC API client was exiting due to
authentication failure exception. Changed code to handle all
exceptions and keep on trying to create the API client. The
node status will show the API connection down in case we are
not able to create the VNC API client.

Closes-Bug: #1611158
(cherry picked from commit 8072aa5ffd37e4082d7ae9697020a6160e8d2682)

6. Change the obj-perms API to pass in the user token in HTTP headers

With PKI tokens, when user token was passed in query parameters for
obj-perms API the token was getting truncated. Changed the API
to accept user token in X-USER-TOKEN HTTP header.

Closes-Bug: #1614376

Conflicts:
 src/config/api-server/tests/test_perms2.py

7. Fix issue with retrieving the db usage info in analytics-api

Closes-Bug: #1614285
(cherry picked from commit 0ec8bf74ba106d655b9a72398f0c9380c2755497)

Change-Id: Id715e40fe3996964b5298da1cd63c248243071dd

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22536
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22536
Committed: http://github.org/Juniper/contrail-controller/commit/761ffd96941cd9ec2f670675fbe553080c4790ec
Submitter: Zuul
Branch: R3.0

commit 761ffd96941cd9ec2f670675fbe553080c4790ec
Author: Megh Bhatt <email address hidden>
Date: Wed Jun 8 18:21:34 2016 -0700

1. Add option for cloud admin access only for analytics REST API

Allow cloud admin role access only for analytics REST API controlled
via --cloud_admin_access_only currently defaulted to False but will default
to True once provisioning changes are done. contrail-analytics-api will
validate role from the X-Auth-Token header via vnc_api/contrail-api. For
debug/administration a localhost bound port 8181 - --admin_port is provided
that requires basic HTTP access authentication.

Clients of analytics REST API - contrail-flows, contrail-logs, contrail-stats,
contrail-topology are changed to use admin port. contrail-svc-monitor is changed
to use auth token.

Conflicts:
 src/opserver/SConscript

Partial-Bug: #1461175
(cherry picked from commit 5492f71383123fea8240ca265e125aee28d9349f)

2. Rename cloud_admin_access_only to multi_tenancy in contrail-analytics-api

Closes-Bug: #1461175
(cherry picked from commit 36df0991a47068bcb6af8cd219e416e2ca60d4cd)

3. for bool option, a conversion from string to bool is required.
Closes-Bug: #1595044

(cherry picked from commit 1d6b81bccf5a7aee39fbb60bd25152e1b8726206)

4. Change cloud admin role name to "cloud-admin" from "admin" for
analytics API access

Closes-Bug: #1600699
(cherry picked from commit 8c131016252a22c52cdfab8042571598818f82c3)

5. Rename multi_tenancy to aaa_mode for analytics API

Handle keystone v2 and v3 token infos returned by
VNC API. Enable cloud-admin-only aaa_mode by default

Change analytics DB and underlay to overlay mapper to
use local admin port when quering opserver

Do not cache auth_token in vnc lib

Closes-Bug: #1599654
(cherry picked from commit a2a7c9248b3d9830d491ab6baf7d21bd9aa64ff6)

6. Changes to bring analytics authenticated access in sync with config

1. Rename aaa_mode value cloud-admin-only to cloud-admin
2. CLOUD_ADMIN_ROLE defaults to admin instead of cloud-admin

Partial-Bug: #1607563
(cherry picked from commit 42db6e38e55bc2410297a99c2af3bea03faa938c)

7. Fix missing import of OpServerUtils in analytics_db.py

Closes-Bug: #1609054
(cherry picked from commit cf5f0567c9bb03e83cd83515b775d2018e668d0c)

8. Remove aaa_mode value cloud-admin-only

Closes-Bug: #1609987

9. Keep on trying to create VNC API client from analytics API

The gevent that creates the VNC API client was exiting due to
authentication failure exception. Changed code to handle all
exceptions and keep on trying to create the API client. The
node status will show the API connection down in case we are
not able to create the VNC API client.

Closes-Bug: #1611158
(cherry picked from commit 8072aa5ffd37e4082d7ae9697020a6160e8d2682)

10. Keystone middleware doesn't like if token is unicode. It must be converted
to string before validation.

Fixes-Bug: #1604773
(cherry picked from commit 18df64367eb5468bbca403aef4f2d22d02be4636)

11. Change the obj-perms API to pass in the user token in HTTP headers

...