Some inherited projects missing when listing user's projects

[Martin Chlumsky]

When a project is added as a child to another project and a user has an inherited role as well as an explicit role on that parent project, the child project may not appear when the user lists their projects.

It appears that the order in which the inherited and effective role assignments are made makes a difference.

What actually happens:

# The parent
$ openstack project show parent --children
+-------------+--------------------------------------------+
| Field | Value |
+-------------+--------------------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | da2265680b3844eaa241a14ac9ee07f1 |
| is_domain | False |
| name | parent |
| parent_id | default |
| subtree | {'3e5e4084c9984d55935198eed49f7164': None} |
| tags | [] |
+-------------+--------------------------------------------+

# A first child
$ openstack project show 3e5e4084c9984d55935198eed49f7164
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | 3e5e4084c9984d55935198eed49f7164 |
| is_domain | False |
| name | child |
| parent_id | da2265680b3844eaa241a14ac9ee07f1 |
| tags | [] |
+-------------+----------------------------------+

# Next, we give user mradmin the project_admin role on the parent project explicitly.
$ openstack role add --project parent --user mradmin project_admin

# We give user mradmin the project_admin role on the parent project's subtree via inheritance.
$ openstack role add --project parent --user mradmin --inherited project_admin

# When we list the projects as user mradmin, everything is fine for now.
$ openstack project list
+----------------------------------+--------+
| ID | Name |
+----------------------------------+--------+
| 3e5e4084c9984d55935198eed49f7164 | child |
| da2265680b3844eaa241a14ac9ee07f1 | parent |
+----------------------------------+--------+

* Important note: the first child project exists before we do the role assignments. The second child project is added after the role assignments.

# Add a second child project to the parent project:
$ openstack project create --parent parent child2
+-------------+----------------------------------+
| Field | Value |
+-------------+----------------------------------+
| description | |
| domain_id | default |
| enabled | True |
| id | c781f589110c4d07a96c40b50bc6bd19 |
| is_domain | False |
| name | child2 |
| parent_id | da2265680b3844eaa241a14ac9ee07f1 |
| tags | [] |
+-------------+----------------------------------+

# The second child does not appear when we list the projects as user mradmin
$ openstack project list
+----------------------------------+--------+
| ID | Name |
+----------------------------------+--------+
| 3e5e4084c9984d55935198eed49f7164 | child |
| da2265680b3844eaa241a14ac9ee07f1 | parent |
+----------------------------------+--------+

If we repeat the above except we reverse the order when assigning the project_admin role:
$ openstack role add --project parent --user mradmin --inherited project_admin
$ openstack role add --project parent --user mradmin project_admin

then we are able to see all projects when we list the projects as user mradmin:
$ openstack project list
+----------------------------------+--------+
| ID | Name |
+----------------------------------+--------+
| 79d5300ac137466a9e2a22931d0a6b52 | child2 |
| e18fa9d21fe94bdcb4965233b65081bd | parent |
| e334dcc334804e2888c7146d3a092050 | child |
+----------------------------------+--------+

Expected behavior:
See all child projects regardless of the order of role assignment.

I was able to reproduce this in Queens and Pike.

[Adam Young (ayoung)]

Quite possibly the best bug report I've seen. Thank you.

It sounds like the code to avoid traversing the same path twice needs to be smart enough to identify the differences in the source of the role assignment.

[Martin Chlumsky (martin-chlumsky)]

It appears I have been fooled by caching.

Almost exactly 5 minutes after getting the incomplete list of projects, re-listing the projects yielded a complete list of projects.

I suspect caching because from the source code:

     # TODO(henry-nash): We might want to consider list limiting this at some
     # point in the future.
     @MEMOIZE_COMPUTED_ASSIGNMENTS
     def list_projects_for_user(self, user_id):
         # FIXME(lbragstad): Without the use of caching, listing effective role
         # assignments is slow, especially with large data set (lots of users
         # with multiple role assignments). This should serve as a marker in
         # case we have the opportunity to come back and optimize this code so
         # that it can be performant without having a hard dependency on
         # caching. Please see https://bugs.launchpad.net/keystone/+bug/1700852
         # for more details.

and from our config:
[cache]
expiration_time=600

What I still don't understand is why there appears to be no caching delay when we do the inherited role assignments first.

That being said, I don't think this is a bug anymore. More like a known issue that I didn't know about yesterday.

[Lance Bragstad (lbragstad)]

We can add cache invalidation logic to the project subsystem that invalidates inherited role assignment when creating new projects. We use a similar pattern else where.

[Adam Young (ayoung)]

After additional consideration, we've come to the conclusion that there is likely a cache invalidation we are missing, and that this is likely valid after all.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/581346

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/581346
Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=83e72d74431526b27b8a2f4ac362582a73edea44
Submitter: Zuul
Branch: master

commit 83e72d74431526b27b8a2f4ac362582a73edea44
Author: Sami MAKKI <email address hidden>
Date: Tue Jul 10 14:21:28 2018 +0200

    Invalidate 'computed assignments' cache when creating a project.

    Without it, listing projects results were missing project on which the
    user had an inherited role.

    Change-Id: If8edb3d1d1d3a0dab691ab6c81dd4b42e3b10ab3
    Closes-Bug: #1780159

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/keystone 14.0.0.0rc1 release candidate.