R3.1 Build 2: Cloud admin access to analytics-api broken due to issue with the obj-perms

[Ankit Jain]

1) Seeing Internal Server Error while trying to access analytics-api with the user having cloud admin access

2) UI Monitoring pages are also broken due to the same issue when multi tenancy is enabled in contrail-analytics-api.conf

Steps:
1) Enable multi_tenancy = True in contrail-analytics-api.conf
2) Restart the process
3) Check the UI

OR

Execute the following command with the user having cloud admin access:

curl -s -H "X-Auth-Token: $(keystone token-get | awk '/ id / {print $4}')" nodeg13:8081/analytics/uves/analytics-node/nodeg13

Observation : 1) All analytics nodes shown as down in the UI
              2) Monitoring pages broken
              3) Contrail-analytics-api not accessible

[Biswajit Mandal (bmandal)]

Sent http://10.204.217.53:8081/analytics/uves/control-nodes
With
X-Auth-Token: d7f9656966bd426789a5509f52fdca81
X_API_ROLE: admin

With same token and role, API Server(http://10.204.217.53:8082/virtual-networks) sends correct response, but analytics throws error,

> /usr/lib/python2.7/dist-packages/opserver/opserver.py(409)_impl()
-> user_token = bottle.request.headers.get('X-Auth-Token')
(Pdb)
(Pdb)
(Pdb)
(Pdb)
(Pdb) list
404 @wraps(func)
405 def _impl(self, *f_args, **f_kwargs):
406 if self._args.auth_conf_info.get('cloud_admin_access_only') and \
407 bottle.request.app == bottle.app():
408 import pdb; pdb.set_trace()
409 -> user_token = bottle.request.headers.get('X-Auth-Token')
410 if not user_token or not \
411 self._vnc_api_client.is_role_cloud_admin(user_token):
412 raise bottle.HTTPResponse(status = 401,
413 body = 'Authentication required',
414 headers = self._reject_auth_headers())
(Pdb) p bottle.request.headers.__dict__
{'environ': {'SERVER_SOFTWARE': 'gevent/1.0 Python/2.7', 'SCRIPT_NAME': '', 'REQUEST_METHOD': 'GET', 'PATH_INFO': '/analytics/uves/control-nodes', 'SERVER_PROTOCOL': 'HTTP/1.1', 'QUERY_STRING': '', 'bottle.app': <bottle.Bottle object at 0x7f270aa57690>, 'REMOTE_ADDR': '172.29.235.139', 'HTTP_X_AUTH_TOKEN': 'd7f9656966bd426789a5509f52fdca81', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36', 'HTTP_CONNECTION': 'keep-alive', 'SERVER_NAME': 'localhost', 'REMOTE_PORT': '51182', 'wsgi.url_scheme': 'http', 'bottle.request': <LocalRequest: GET http://10.204.217.53:8081/analytics/uves/control-nodes>, 'SERVER_PORT': '8081', 'bottle.request.headers': <bottle.WSGIHeaderDict object at 0x7f26ff65a990>, 'route.handle': <GET '/analytics/uves/<tables>' <function dyn_list_http_get at 0x7f270524be60>>, 'bottle.request.urlparts': SplitResult(scheme='http', netloc='10.204.217.53:8081', path='/analytics/uves/control-nodes', query='', fragment=''), 'route.url_args': {'tables': 'control-nodes'}, 'wsgi.input': <gevent.pywsgi.Input object at 0x7f26ff6f4110>, 'HTTP_HOST': '10.204.217.53:8081', 'wsgi.multithread': False, 'HTTP_CACHE_CONTROL': 'no-cache', 'HTTP_ACCEPT': '*/*', 'bottle.raw_path': '/analytics/uves/control-nodes', 'wsgi.version': (1, 0), 'bottle.route': <GET '/analytics/uves/<tables>' <function dyn_list_http_get at 0x7f270524be60>>, 'GATEWAY_INTERFACE': 'CGI/1.1', 'wsgi.run_once': False, 'wsgi.errors': <open file '<stderr>', mode 'w' at 0x7f270e4281e0>, 'wsgi.multiprocess': False, 'HTTP_ACCEPT_LANGUAGE': 'en-US,en;q=0.8', 'HTTP_X_API_ROLE': 'admin', 'HTTP_ACCEPT_ENCODING': 'gzip, deflate, sdch'}}
(Pdb) n
> /usr/lib/python2.7/dist-packages/opserver/opserver.py(410)_impl()
-> if not user_token or not \
(Pdb) p user_token
'd7f9656966bd426789a5509f52fdca81'
(Pdb) n
> /usr/lib/python2.7/dist-packages/opserver/opserver.py(411)_impl()
-> self._vnc_api_client.is_role_cloud_admin(user_token):
(Pdb) p user_token
'd7f9656966bd426789a5509f...

[Megh Bhatt (meghb)]

This is a contrail-api issue with the obj-perms API which is used by contrail-analytics-api:

WARNING:keystonemiddleware.auth_token:Use of the auth_admin_prefix, auth_host, auth_port, auth_protocol, identity_uri, admin_token, admin_user, admin_password, and admin_tenant_name configuration options was deprecated in the Mitaka release in favor of an auth_plugin and its related options. This class may be removed in a future release.
WARNING:keystonemiddleware.auth_token:Configuring admin URI using auth fragments was deprecated in the Kilo release, and will be removed in the N release, use 'identity_uri\ instead.
WARNING:keystonemiddleware.auth_token:Configuring auth_uri to point to the public identity endpoint is required; clients may not be able to authenticate against an admin endpoint
WARNING:keystonemiddleware.auth_token:Authorization failed for token
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1618, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1759, in obj_perms_http_get
    token_info = self._auth_svc.validate_user_token(get_request())
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_auth_keystone.py", line 236, in validate_user_token
    return auth_middleware(request.headers.environ, None)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
    response = req.get_response(self._app)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
    output.extend(app_iter)
TypeError: 'NoneType' object is not iterable
10.204.217.53 - - [2016-07-20 23:15:11] "GET /obj-perms?token=b81efadaf55e4c14937365eda5df4914 HTTP/1.1" 500 156 0.019420
ERROR:contrail-api:__default__ [SYS_ERR]: VncApiError: <type 'exceptions.TypeError'>
Python 2.7.6: /usr/bin/python
Wed Jul 20 23:15:11 2016

A problem occurred in a Python script. Here is the sequence of
function calls leading up to the error, in the order they occurred.

 /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py in handler_trap_exception(*args=(), **kwargs={})
 1616 (code, err_msg) = status
 1617 raise cfgm_common.exceptions.HttpError(code, err_msg)
 1618 response = handler(*args, **kwargs)
 1619 self._generate_rest_api_response_trace(trace, response)
 1620
response undefined
handler = <bound method VncApiServer.obj_perms_http_get of...i_server.vnc_cfg...

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22234
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22276
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22234
Committed: http://github.org/Juniper/contrail-controller/commit/58b7add704c47973ace3690b10fa58acd76b5bb1
Submitter: Zuul
Branch: master

commit 58b7add704c47973ace3690b10fa58acd76b5bb1
Author: Deepinder Setia <email address hidden>
Date: Wed Jul 20 15:25:37 2016 -0700

Keystone middleware doesn't like if token is unicode. It must be converted
to string before validation.

Change-Id: I537f3ca96229fd9abef18d8ceb56f3fee6fd1296
Fixes-Bug: #1604773

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22396
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22276
Committed: http://github.org/Juniper/contrail-controller/commit/18df64367eb5468bbca403aef4f2d22d02be4636
Submitter: Zuul
Branch: R3.1

commit 18df64367eb5468bbca403aef4f2d22d02be4636
Author: Deepinder Setia <email address hidden>
Date: Wed Jul 20 15:25:37 2016 -0700

Keystone middleware doesn't like if token is unicode. It must be converted
to string before validation.

Change-Id: I537f3ca96229fd9abef18d8ceb56f3fee6fd1296
Fixes-Bug: #1604773

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22536
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22396
Committed: http://github.org/Juniper/contrail-controller/commit/76881c37eab1ebda95ed85f37a86d0d0e37a9755
Submitter: Zuul
Branch: R3.0

commit 76881c37eab1ebda95ed85f37a86d0d0e37a9755
Author: Deepinder Setia <email address hidden>
Date: Wed Jul 20 15:25:37 2016 -0700

Keystone middleware doesn't like if token is unicode. It must be converted
to string before validation.

Change-Id: I537f3ca96229fd9abef18d8ceb56f3fee6fd1296
Fixes-Bug: #1604773

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22536
Submitter: Megh Bhatt (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22536
Committed: http://github.org/Juniper/contrail-controller/commit/761ffd96941cd9ec2f670675fbe553080c4790ec
Submitter: Zuul
Branch: R3.0

commit 761ffd96941cd9ec2f670675fbe553080c4790ec
Author: Megh Bhatt <email address hidden>
Date: Wed Jun 8 18:21:34 2016 -0700

1. Add option for cloud admin access only for analytics REST API

Allow cloud admin role access only for analytics REST API controlled
via --cloud_admin_access_only currently defaulted to False but will default
to True once provisioning changes are done. contrail-analytics-api will
validate role from the X-Auth-Token header via vnc_api/contrail-api. For
debug/administration a localhost bound port 8181 - --admin_port is provided
that requires basic HTTP access authentication.

Clients of analytics REST API - contrail-flows, contrail-logs, contrail-stats,
contrail-topology are changed to use admin port. contrail-svc-monitor is changed
to use auth token.

Conflicts:
 src/opserver/SConscript

Partial-Bug: #1461175
(cherry picked from commit 5492f71383123fea8240ca265e125aee28d9349f)

2. Rename cloud_admin_access_only to multi_tenancy in contrail-analytics-api

Closes-Bug: #1461175
(cherry picked from commit 36df0991a47068bcb6af8cd219e416e2ca60d4cd)

3. for bool option, a conversion from string to bool is required.
Closes-Bug: #1595044

(cherry picked from commit 1d6b81bccf5a7aee39fbb60bd25152e1b8726206)

4. Change cloud admin role name to "cloud-admin" from "admin" for
analytics API access

Closes-Bug: #1600699
(cherry picked from commit 8c131016252a22c52cdfab8042571598818f82c3)

5. Rename multi_tenancy to aaa_mode for analytics API

Handle keystone v2 and v3 token infos returned by
VNC API. Enable cloud-admin-only aaa_mode by default

Change analytics DB and underlay to overlay mapper to
use local admin port when quering opserver

Do not cache auth_token in vnc lib

Closes-Bug: #1599654
(cherry picked from commit a2a7c9248b3d9830d491ab6baf7d21bd9aa64ff6)

6. Changes to bring analytics authenticated access in sync with config

1. Rename aaa_mode value cloud-admin-only to cloud-admin
2. CLOUD_ADMIN_ROLE defaults to admin instead of cloud-admin

Partial-Bug: #1607563
(cherry picked from commit 42db6e38e55bc2410297a99c2af3bea03faa938c)

7. Fix missing import of OpServerUtils in analytics_db.py

Closes-Bug: #1609054
(cherry picked from commit cf5f0567c9bb03e83cd83515b775d2018e668d0c)

8. Remove aaa_mode value cloud-admin-only

Closes-Bug: #1609987

9. Keep on trying to create VNC API client from analytics API

The gevent that creates the VNC API client was exiting due to
authentication failure exception. Changed code to handle all
exceptions and keep on trying to create the API client. The
node status will show the API connection down in case we are
not able to create the VNC API client.

Closes-Bug: #1611158
(cherry picked from commit 8072aa5ffd37e4082d7ae9697020a6160e8d2682)

10. Keystone middleware doesn't like if token is unicode. It must be converted
to string before validation.

Fixes-Bug: #1604773
(cherry picked from commit 18df64367eb5468bbca403aef4f2d22d02be4636)

11. Change the obj-perms API to pass in the user token in HTTP headers

...