R3.1 mitaka Build 12: Internal server error when cloud admin user tries to get user token info

[Ankit Jain]

Error seen in contrail-analytics-api-stdout.log

HttpError: HTTP Status: 500 Content: Internal Server Error
10.204.217.53 - - [2016-08-02 12:23:04] "GET /analytics/uves/vrouter/*?cfilt=NodeStatus:process_status HTTP/1.1" 500 156 0.217813
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/opserver/opserver.py", line 411, in _impl
    self._vnc_api_client.is_role_cloud_admin(user_token):
  File "/usr/lib/python2.7/dist-packages/opserver/vnc_cfg_api_client.py", line 64, in is_role_cloud_admin
    result = self._get_user_token_info(user_token)
  File "/usr/lib/python2.7/dist-packages/opserver/vnc_cfg_api_client.py", line 30, in _get_user_token_info
    return self._vnc_api_client.obj_perms(user_token)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 1154, in obj_perms
    rv = self._request_server(rest.OP_GET, "/obj-perms", data=query)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 707, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 770, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 500 Content: Internal Server Error

Error seen in contrail-analytics-api.log
08/02/2016 12:08:48 PM [contrail-analytics-api]: Stopping agguve part 29
08/02/2016 12:08:58 PM [contrail-analytics-api]: Starting agguve part 29 using PartInfo(ip_address=u'10.204.217.53', instance_id=u'0', acq_time=14701194374416
80, port=6379)
08/02/2016 12:09:51 PM [contrail-analytics-api]: <type 'exceptions.NameError'>
08/02/2016 12:09:51 PM [contrail-analytics-api]: ("global name 'OpServerUtils' is not defined",)
08/02/2016 12:09:51 PM [contrail-analytics-api]: global name 'OpServerUtils' is not defined
08/02/2016 12:09:51 PM [contrail-analytics-api]: Could not retrieve db usage information

UI is not able to fetch any data from analytics because of the same issue. Please check.

[Megh Bhatt (meghb)]

If a cloud-admin role was created you will need to change the DEFAULT.cloud_admin_role parameter to cloud-admin in both contrail-analytics-api.conf and contrail-api.conf. We are defaulting cloud_admin_role to admin to maintain backward compatibility.

[Ankit Jain (ankitja)]

Traceback in contrail-api logs:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1632, in handler_trap_exception
    response = handler(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_cfg_api_server.py", line 1773, in obj_perms_http_get
    token_info = self._auth_svc.validate_user_token(get_request())
  File "/usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_auth_keystone.py", line 236, in validate_user_token
    return auth_middleware(request.headers.environ, None)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 467, in __call__
    response = req.get_response(self._app)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1317, in send
    application, catch_exc_info=False)
  File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1292, in call_application
    return (captured[0], captured[1], app_iter)
IndexError: list index out of range

[Ankit Chadha (achadha)]

I'm seeing this one also on an all-in-one setup. The UI reports Analytics-Nodes as 'down'.

root@a4s2:/etc/contrail# cat contrail-api.conf

[DEFAULTS]
ifmap_server_ip=10.84.18.2
ifmap_server_port=8443
ifmap_username=api-server
ifmap_password=api-server
cassandra_server_list=10.84.18.2:9160
listen_ip_addr=0.0.0.0
listen_port=8082
log_file=/var/log/contrail/contrail-api.log
log_local=1
log_level=SYS_NOTICE
disc_server_ip=10.84.18.2
disc_server_port=5998
zk_server_ip=10.84.18.2:2181
rabbit_server=10.84.18.2:5672
list_optimization_enabled=True
auth = keystone

[SECURITY]
use_certs=False
keyfile=/etc/contrail/ssl/private_keys/apiserver_key.pem
certfile=/etc/contrail/ssl/certs/apiserver.pem
ca_certs=/etc/contrail/ssl/certs/ca.pem
root@a4s2:/etc/contrail#

root@a4s2:/etc/contrail# cat contrail-analytics-api.conf
[DEFAULTS]
#host_ip = 127.0.0.1
#collectors = 127.0.0.1:8086
#cassandra_server_list=127.0.0.1:9160
#http_server_port = 8090
#rest_api_port = 8081
#rest_api_ip = 0.0.0.0
log_local = 1
log_level = SYS_NOTICE
#log_category =
log_file = /var/log/contrail/contrail-analytics-api.log
# Sandesh send rate limit can be used to throttle system logs transmitted per
# second. System logs are dropped if the sending rate is exceeded
#sandesh_send_rate_limit = 100
partitions=30
cassandra_server_list = 10.84.18.2:9042
api_server = 10.84.18.2:8082
log_category =
http_server_port = 8090
host_ip = 10.84.18.2
analytics_data_ttl = None
aaa_mode = cloud-admin-only
analytics_statistics_ttl = None
analytics_config_audit_ttl = None
analytics_flow_ttl = None
rest_api_port = 8081

[DISCOVERY]
disc_server_port = 5998
disc_server_ip = 10.84.18.2
#disc_server_ip = 127.0.0.1
#disc_server_port = 5998

[REDIS]
redis_server_port = 6379
redis_query_port = 6379
#server=127.0.0.1
#redis_server_port=6379
#redis_query_port=6379

root@a4s2:/etc/contrail#

Version: 3.1.0.0-13~mitaka

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/22782
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/22782
Committed: http://github.org/Juniper/contrail-controller/commit/fa9077dbc4ae0077f4a41df51fd6d37b773496bb
Submitter: Zuul
Branch: R3.1

commit fa9077dbc4ae0077f4a41df51fd6d37b773496bb
Author: Deepinder Setia <email address hidden>
Date: Tue Aug 2 16:49:45 2016 -0700

Change invocation of call to keystone middleware for token validation
that meets Mitaka expectations.

Change-Id: I98bfcec13eba314504c81c45f927cee36f0d5234
Closes-Bug: 1608815

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23316
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23316
Committed: http://github.org/Juniper/contrail-controller/commit/7be5fa8b310012626483f31aa7ef11b0bea01907
Submitter: Zuul
Branch: R3.0

commit 7be5fa8b310012626483f31aa7ef11b0bea01907
Author: Deepinder Setia <email address hidden>
Date: Tue Aug 2 16:49:45 2016 -0700

Change invocation of call to keystone middleware for token validation
that meets Mitaka expectations.

Closes-Bug: 1608815

Conflicts:
 src/config/api-server/vnc_auth_keystone.py

Change-Id: I69c69c0c9a88cb27e818bdae59289ea24d8ac7bf

[OpenContrail Admin (ci-admin-f)]

Review in progress for https://review.opencontrail.org/23708
Submitter: Deepinder Setia (<email address hidden>)

[OpenContrail Admin (ci-admin-f)]

Reviewed: https://review.opencontrail.org/23708
Committed: http://github.org/Juniper/contrail-controller/commit/084b322b0701b51af5871ee9203189d6d2db934a
Submitter: Zuul
Branch: master

commit 084b322b0701b51af5871ee9203189d6d2db934a
Author: Deepinder Setia <email address hidden>
Date: Tue Aug 2 16:49:45 2016 -0700

Change invocation of call to keystone middleware for token validation
that meets Mitaka expectations.

Closes-Bug: 1608815

Change-Id: I69c69c0c9a88cb27e818bdae59289ea24d8ac7bf