Split backend crashes with AttributeError

[Otavio Gaspareto]

Using the Keystone split backend code (H2) with both identity and assignment using SQL drivers, during the process of authentication thru Horizon, I’m getting the following error in the log:

Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 240, in __call__
    result = method(context, **params)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 80, in authenticate
    context, auth)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 257, in _authenticate_local
    user_id, tenant_id)
  File "/usr/lib/python2.6/site-packages/keystone/token/controllers.py", line 371, in _get_project_roles_and_ref
    user_id, tenant_id)
  File "/usr/lib/python2.6/site-packages/keystone/identity/core.py", line 123, in get_roles_for_user_and_project
    user_id, tenant_id)
  File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 125, in get_roles_for_user_and_project
    user_role_list = _get_user_project_roles(user_id, project_ref)
  File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 107, in _get_user_project_roles
    metadata_ref.get('roles', {}), False)
  File "/usr/lib/python2.6/site-packages/keystone/common/manager.py", line 44, in _wrapper
    return f(*args, **kw)
  File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 216, in _roles_from_role_dicts
    if ((not d.get('inherited_to') and not inherited) or
AttributeError: 'unicode' object has no attribute 'get'

[Otavio Gaspareto (otavio-barcelos-gaspareto-deactivatedaccount)]

I could bypass this problem with this code at keystone/assignment/core.py:

    def _roles_from_role_dicts(self, dict_list, inherited):
        role_list = []
        for d in dict_list:
            # To bypass the problem
            if isinstance(d, unicode):
                d = {'id':d}
            #
            if ((not d.get('inherited_to') and not inherited) or
               (d.get('inherited_to') == 'projects' and inherited)):
                role_list.append(d['id'])
        return role_list

[Adam Young (ayoung)]

 Adding in
           if isinstance(d, unicode):
                d = {'id':d}
Seems suspect. It means that something was supposed to be a dictionary, but showed up as unicode. I would expect you to have to convert to string and then to JSON.

[Dolph Mathews (dolph)]

Client side:

REQ: curl -i http://localhost:35357/v3/projects/71c11ba9423e40b48bdffb8e649d4012/users/85973994c2c146d0bbc19849a6710c44/roles/9bb1083cf50a4589ac12776ffe586233 HEAD -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: ADMIN"
RESP: [500] CaseInsensitiveDict({'date': 'Mon, 12 Aug 2013 17:47:47 GMT', 'vary': 'X-Auth-Token', 'content-length': '183', 'content-type': 'application/json'})
RESP BODY:

No body was returned.
Request returned failure status: 500
Traceback (most recent call last):
  File "/Users/dolph/Environments/os/bin/keystone-workout", line 9, in <module>
    load_entry_point('keystone-workout==0.1.0', 'console_scripts', 'keystone-workout')()
  File "/Users/dolph/Projects/keystone-workout/keystoneworkout/cli/core.py", line 59, in cli
    args.func(args)
  File "/Users/dolph/Projects/keystone-workout/keystoneworkout/cli/subcommands.py", line 65, in __call__
    BootstrapAdmin()(args)
  File "/Users/dolph/Projects/keystone-workout/keystoneworkout/cli/subcommands.py", line 121, in __call__
    role=role)
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/v3/roles.py", line 125, in check
    role_id=base.getid(role))
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/base.py", line 73, in func
    return f(*args, **kwargs)
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/base.py", line 254, in head
    return self._head(self.build_url(dict_args_in_out=kwargs))
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/base.py", line 109, in _head
    resp, body = self.api.head(url)
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/client.py", line 611, in head
    return self._cs_request(url, 'HEAD', **kwargs)
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/client.py", line 604, in _cs_request
    **kwargs)
  File "/Users/dolph/Projects/python-keystoneclient/keystoneclient/client.py", line 576, in request
    raise exceptions.from_response(resp, body or resp.text)
keystoneclient.exceptions.ClientException: <attribute 'message' of 'exceptions.BaseException' objects> (HTTP 500)

service side:

2013-08-12 12:47:47 ERROR [keystone.common.wsgi] 'unicode' object has no attribute 'get'
Traceback (most recent call last):
  File "/Users/dolph/Projects/keystone/keystone/common/wsgi.py", line 240, in __call__
    result = method(context, **params)
  File "/Users/dolph/Projects/keystone/keystone/common/controller.py", line 106, in wrapper
    return f(self, context, *args, **kwargs)
  File "/Users/dolph/Projects/keystone/keystone/identity/controllers.py", line 798, in check_grant
    self._check_if_inherited(context))
  File "/Users/dolph/Projects/keystone/keystone/identity/core.py", line 173, in get_grant
    inherited_to_projects))
  File "/Users/dolph/Projects/keystone/keystone/common/manager.py", line 44, in _wrapper
    return f(*args, **kw)
  File "/Users/dolph/Projects/keystone/keystone/assignment/backends/sql.py", line 180, in get_grant
    metadata_ref.get('roles', []), inherited_to_projects))
  File "/Users/dolph/Projects/keystone/keystone/assignment/core.py", line 224, in _roles_from_role_dicts
    if ((not d.get('inherited_t...

[Dolph Mathews (dolph)]

Raising priority as basic v3 role assignments appear to be completely unusable (?).

[Dolph Mathews (dolph)]

Cross-linking... after reproducing this bug, I produced a different backtrace that may share the same underlying cause... see bug 1211445

[Henry Nash (henry-nash)]

So having trouble reproducing this. Used a similar curl test to the one the one Dolphm posted, and no errors. Could you share your config file in terms of the various drivers for each backend?

Also, was the DB migrated from a a pre-H2 version, or set up new?

[Otavio Gaspareto (otavio-barcelos-gaspareto-deactivatedaccount)]

Henry, answering your question #6, I didn't use any DB migration script. I tested this using a pre-H2 version environment.
In fact, there are any DB migration script that are mandatory be fired in my environment before start to use the split backend?

[Adam Young (ayoung)]

Otavio, I would at least ask that you make sure your database is up to the latest version by running keystone-manage db-sync

[Dolph Mathews (dolph)]

Downgrading priority as I'm unable to reproduce on master (i.e. if the issue still exists - it's doesn't appear to be a critical blocker).

[Dolph Mathews (dolph)]

Closing, as I haven't been able to reproduce this at all recently. If it is still reproducible on master, please comment here! Thanks.