policy.v3cloudsample.json broken in mitaka

[Jay Jahns]

We have a multi-domain configuration in our private cloud that I've had to revert to using the Liberty policy.v3cloudsample.json file instead of Mitaka or master.

Horizon is generating the following trace when a domain admin is trying to look at projects/users:

[pid: 22842|app: 0|req: 5/17] 10.38.202.12 () {46 vars in 907 bytes} [Thu Jun 2 07:17:24 2016] GET / => generated 0 bytes in 5 msecs (HTTP/1.1 302) 5 headers in 198 bytes (1 switches on core 1)
Internal Server Error: /identity/
Traceback (most recent call last):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get
    handled = self.construct_tables()
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables
    handled = self.handle_table(table)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table
    data = self._get_data_dict()
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/projects/views.py", line 84, in get_data
    self.request):
  File "/opt/mhos/openstack/horizon/openstack_dashboard/policy.py", line 24, in check
    return policy_check(actions, request, target)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 155, in check
    enforcer[scope], action, target, domain_credentials)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 169, in _check_credentials
    if not enforcer_scope.enforce(action, target, credentials):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 578, in enforce
    result = self.rules[rule](target, creds, self)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__
    if rule(target, cred, enforcer):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 204, in __call__
    return enforcer.rules[self.match](target, creds, enforcer)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 125, in __call__
    if not rule(target, cred, enforcer):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__
    if rule(target, cred, enforcer):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 311, in __call__
    return self._find_in_dict(creds, path_segments, match)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 292, in _find_in_dict
    return self._find_in_dict(test_value, path_segments, match)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 283, in _find_in_dict
    test_value = test_value[key]
TypeError: 'Token' object has no attribute '__getitem__'
[pid: 22837|app: 0|req: 5/18] 10.38.202.12 () {46 vars in 925 bytes} [Thu Jun 2 07:17:24 2016] GET /identity/ => generated 375516 bytes in 251 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0)

Or we will get another trace, as follows, which is a bit more understanding:

[pid: 22623|app: 0|req: 17/76] 10.38.202.12 () {44 vars in 3206 bytes} [Thu Jun 2 07:05:15 2016] GET /i18n/js/horizon+openstack_dashboard+neutron_lbaas_dashboard+muranodashboard/ => generated 2372 bytes in 4 msecs (HTTP/1.1 200) 4 hea
ders in 132 bytes (1 switches on core 1)
Pure project admin doesn't have a domain token
Internal Server Error: /identity/users/
Traceback (most recent call last):
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get
    handled = self.construct_tables()
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables
    handled = self.handle_table(table)
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table
    data = self._get_data_dict()
  File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict
    self._data = {self.table_class._meta.name: self.get_data()}
  File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/users/views.py", line 79, in get_data
    u.domain_name = domain_lookup.get(u.domain_id)
AttributeError: 'NoneType' object has no attribute 'get'
[pid: 22619|app: 0|req: 15/77] 10.38.202.12 () {46 vars in 3190 bytes} [Thu Jun 2 07:05:19 2016] GET /identity/users/ => generated 340688 bytes in 413 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0)

The 2nd trace usually is associated with situations where the V2 policy is in effect.

Horizon Config:
Memcached backend for caching
Session DB configured

As a workaround, we reverted to the Liberty policy.v3cloudsample.json and multi-domain is working beautifully, and our domain admins are able to manage users,projects,roles of the domain.

We believe the issue lies around the following line (pulled from master):
https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json#L3

And yes, the admin_domain_id was correctly set.

We feel that this breaks domain functionality and I would like someone to take a look and let us know if this is a configuration problem and why this works without failure with the Liberty policy.

[Dolph Mathews (dolph)]

I'm trying to read between the backtrace lines, and I actually can't tell if "Pure project admin doesn't have a domain token" is part of an error message or not - can you clarify? I don't see anything in keystone that would raise that, but it sounds like something Adam would know about :)

I also don't understand when you get the second backtrace - it looks like a user object is unexpectedly (and wrongly) missing a domain_id attribute (that should never happen). The first is a failure in policy, again, that I'm hoping Adam Young can explain.

[Marc Heckmann (marc-w-heckmann)]

I've done some debugging on this and while I don't get the backtrace, I can confirm that domain admins I have setup aren't able to list projects (or do anything else) on the domain because the "domain_id" attribute us missing from the creds being evaluated in the policy enforcement.

[Marc Heckmann (marc-w-heckmann)]

I forget to mention in my previous comment that in my case, it's the "admin_and_matching_domain_id" rule from the v3 sample policy which is broken.

[Marc Heckmann (marc-w-heckmann)]

So further debugging of my case shows that this happens only when Domain Admin user is assigned admin role on the domain AND a project within the domain.

If I remove the admin role assignment on the project, all works as expected.

Is this intended behaviour? a bug?

[Adam Young (ayoung)]

I think this is a Horizon bug, not Keystone. The stack trace is all Horizon code.

I suspect it is a conflict between domain and project scoped token code in Horizon

[Steve Martinelli (stevemar)]

Looking at the two default policy files, i'm not sure what is causing the regression:

for liberty, the "admin_required" rule evaluates to: "admin_required": "role:admin"

liberty: "cloud_admin": "rule:admin_required and domain_id:admin_domain_id",
mitaka: "cloud_admin": "role:admin and (token.is_admin_project:True or domain_id:admin_domain_id)",

And since the extra stuff related to "token.is_admin_project:True" is in an OR, that change should be backwards compatible.

[Steve Martinelli (stevemar)]

We'd need to see the response coming back from Keystone to see if it's malformed. It could be that horizon is making assumptions about how the response looks

[Brad Pokorny (bpokorny)]

I think this is a duplicate of https://bugs.launchpad.net/oslo.policy/+bug/1547684. After this line in the stack trace, the stack traces are identical except for line numbers that may have changed since 1547684 was created:

  File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/projects/views.py", line 84, in get_data
    self.request):

Similar to the workaround for this one, we got around 1547684 by switching to the Liberty policy file as well.