keystone v3 :: permission denied error while viewing project.

[Ritam Gangopadhyay]

after logging in with new user credentials in a newly created domain the Dashboard, Projects and Networks tab under Networking doesn't show any data. In logs we see a "Permission Denied" error.

Login:-
Username - U_DM_1
Paswd - udm1
Domain - DM_1

Setup:- This is a HA setup.

build 2738 - liberty

host1 = 'root@10.204.221.24'
host2 = 'root@10.204.221.27'
host3 = 'root@10.204.221.28'
host4 = 'root@10.204.221.25'
host5 = 'root@10.204.221.26'

env.hostnames ={
    'all': ['nodeg34', 'nodec48', 'nodec49', 'nodec51', 'nodec63']
}

env.roledefs = {

    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1, host2, host3],
    'openstack':[host1, host2, host3],
    'control':[host1, host2, host3],
    'compute': [host4, host5],
    'collector': [host1, host2, host3],
    'webui': [host1, host2, host3],
    'database': [host1, host2, host3],
    'build': [host_build],

}

/var/log/contrail/contrail-webui-stdout.log

06/29/2016 11:02:42 PM - ^[[31merror^[[39m: URL [http://192.168.100.6:9100/domain/2766adff-5aa5-40ca-ae29-f2c7bdfaa34d] returned error ["Permission Denied"]
06/29/2016 11:02:42 PM - ^[[31merror^[[39m: Encountered broken link: /domain/2766adff-5aa5-40ca-ae29-f2c7bdfaa34d
06/29/2016 11:02:42 PM - ^[[31merror^[[39m: REST Server Error: Permission Denied
    at APIServer.retryMakeCall (/usr/src/contrail/contrail-web-core/src/serverroot/common/rest.api.js:180:13)
    at Request.<anonymous> (/usr/src/contrail/contrail-web-core/src/serverroot/common/rest.api.js:312:18)
    at Request.emit (events.js:98:17)
    at Request.mixin._fireSuccess (/usr/lib64/node_modules/restler/lib/restler.js:226:10)
    at /usr/lib64/node_modules/restler/lib/restler.js:157:20
    at IncomingMessage.parsers.auto (/usr/lib64/node_modules/restler/lib/restler.js:390:7)
    at Request.mixin._encode (/usr/lib64/node_modules/restler/lib/restler.js:194:29)
    at /usr/lib64/node_modules/restler/lib/restler.js:153:16
    at Request.mixin._decode (/usr/lib64/node_modules/restler/lib/restler.js:169:7)
    at IncomingMessage.<anonymous> (/usr/lib64/node_modules/restler/lib/restler.js:146:14)
Sending requrl as: http://192.168.100.5:9100/domains undefined { 'X-Auth-Token': '069c5cb1c3b5443da018ad2d515fc747',
  X_API_ROLE: 'admin',
  'Content-Length': 0 }
06/29/2016 11:02:42 PM - ^[[31merror^[[39m: URL [http://192.168.100.5:9100/domains] returned error ["Permission Denied"]

[Ritam Gangopadhyay (ritam)]

Screen Shot of the error.

[Biswajit Mandal (bmandal)]

> /usr/lib/python2.7/dist-packages/vnc_cfg_api_server/vnc_rbac.py(206)validate_request()
-> return (False, err_msg)
(Pdb)
(Pdb)
(Pdb)
(Pdb) p request
<LocalRequest: GET http://192.168.100.5:8082/global-vrouter-configs?detail=true>
(Pdb) p request.__dict__
{}
(Pdb) p domain_id
None
(Pdb) p rule_list
[]
(Pdb) c
192.168.100.5 - - [2016-07-03 04:20:29] "GET /global-vrouter-configs?detail=true HTTP/1.1" 403 140 40.608721
WARNING:contrail-api:__default__ [SYS_NOTICE]: VncApiError: rbac: rule list empty!!
192.168.100.7 - - [2016-07-03 04:20:29] "GET /virtual-routers?detail=True HTTP/1.1" 200 2258 0.058668
WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers
192.168.100.6 - - [2016-07-03 04:20:37] "GET / HTTP/1.1" 401 197 0.000495
192.168.100.6 - - [2016-07-03 04:20:37] "GET / HTTP/1.1" 200 17994 0.074802
192.168.100.6 - - [2016-07-03 04:20:37] "GET /physical-routers?count=False&detail=False HTTP/1.1" 200 132 0.019422
WARNING:keystoneclient.middleware.auth_token:Unable to find authentication token in headers

From UI Sent Headers:
http://192.168.100.5:8082/global-vrouter-configs?detail=true { 'X-Auth-Token': '31c96dc6309d49318f45bf581c333db8',
  X_API_ROLE: 'admin',
  'Content-Length': 0 }

[Deepinder Setia (dsetia)]

Ritam, listing project and users requires domain scoped token. Is that what is being used here?

  "cloud_admin": "role:admin and (token.is_admin_project:True or domain_id:default)",
 "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id",

Note the domain requirement in cloud_admin clause.

[Deepinder Setia (dsetia)]

Ritam, can you retry and let me know if this is still an issue?

[Ritam Gangopadhyay (ritam)]

Reason for this issue is a setting in horizon local_setting.py file where we can set memcached to use cache instead of cookies the authentication token in case of mitaka is quite large and cookies don't support it after making the required changes and restarting horizon and keystone this issue was fix verified on build 16 mitaka.

Deepinder has incorporated this change in his v3 provisioning automation which is supposed to come in build 18 and was merged last night. Once I verify in build 18 I will close the bug.