Slow listing projects for user with many role assignments

[Steve McLellan]

With a large number of role assignments (e.g 500) it becomes very slow to list the projects a user has access to (via /users/<id>/projects). I'm seeing times of around 4 seconds versus 0.1 for a user with a couple of assignments.

Instrumenting list_projects_for_user (https://github.com/openstack/keystone/blob/stable/newton/keystone/assignment/core.py#L268), where each number is the time elapsed since the start of list_projects_for_user, I get:

  list_projects_for_user 3.998 role_assignments
  list_projects_for_user 3.999 project_ids
  list_projects_for_user 4.105 list_projects

The time is spent on the call made to list_role_assignments on line 269 which in turns calls _list_effective_role_assignments at https://github.com/openstack/keystone/blob/stable/newton/keystone/assignment/core.py#L986.

Listing role assignments for a user directly (with GET /role_assignments?user.id=<id>) is very fast, which is further indication that something in the effective role processing is the cause. I haven't yet timed the internal of _list_effective_role_assignments.

Running keystone mitaka (though I believe this would apply to master) with users in LDAP but roles and projects managed by keystone.

[Lance Bragstad (lbragstad)]

This seems like something we should be able to fix and backport to Ocata since it has an impact on performance.

[Steve McLellan (sjmc7)]

Looks like the slow code here is strip_domain_roles:

    def _strip_domain_roles(self, role_refs):
        """Post process assignment list for domain roles.

        Domain roles are only designed to do the job of inferring other roles
        and since that has been done before this method is called, we need to
        remove any assignments that include a domain role.

        """
        def _role_is_global(role_id):
            ref = self.role_api.get_role(role_id)
            return (ref['domain_id'] is None)

        filter_results = []
        for ref in role_refs:
            if _role_is_global(ref['role_id']):
                filter_results.append(ref)
        return filter_results

With caching the time would probably go away, but as it is, it's making a DB lookup for each role.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/487143

[Lance Bragstad (lbragstad)]

Steve, I've proposed a patch that implements caching for the API in question [0]. Let me know if that helps in any way. It might end up being a work-around until we can get some patches proposed to speed up the assignment API.

[0] https://review.openstack.org/#/c/487143/

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/491921

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/491921
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=9fccd38d1b177664c64bbf987a8c31d06d7cfc60
Submitter: Jenkins
Branch: master

commit 9fccd38d1b177664c64bbf987a8c31d06d7cfc60
Author: Lance Bragstad <email address hidden>
Date: Tue Aug 8 20:37:10 2017 +0000

    Remove unused hints from assignment APIs

    The controller is responsible for listing user projects based on role
    assignments and would build a hints objects and pass it to the
    assignment manager. This is a common pattern used across keystone's
    APIs. But, the assignment API never actually passed the hints objects
    to the backend implementation.

    This commit removes the hints from being passed to the manager for
    list_projects_for_user and list_domains_for_user because those
    APIs never use the hints object. This should allow us to implement
    caching to speed up those calls later.

    Change-Id: I9b1c8c30ca6a78dd6e78add7de278e467ceea046
    Related-Bug: 1700852

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/487143
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=63124f703a81074793360c1b91711b6ee5a76196
Submitter: Jenkins
Branch: master

commit 63124f703a81074793360c1b91711b6ee5a76196
Author: Lance Bragstad <email address hidden>
Date: Tue Jul 25 17:03:55 2017 +0000

    Cache list projects and domains for user

    Listing projects and domains for a user based on their role
    assignments was noted as being really slow, especially when users
    have a lot of assignments. This commit implements caching to mitigate
    the issue while we continue to investigate ways to speed up the
    assignment API.

    Change-Id: I72e398c65f01aa4f9a37f817d184a13ed01089ce
    Closes-Bug: 1700852

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 12.0.0.0rc1 release candidate.