[<= Queens] With token-provider='uuid', roles of dynamically obtained federated groups are not taken into account during token-based authentication (for project-scoped token creation)

[Dmitrii Shcherbakov]

[Overview]
The relevant part of the federated authentication process after the IdP and SP token parsing stages is as follows:

1) WSGI environment variables created based on token attributes (e.g. SAML token attributes) are passed down to Keystone;
2) Keystone creates a shadow mapped user in the db and tries to map token attributes to objects such as groups, roles and projects in the DB based on a custom mapping created by an operator;
3) groups that may be obtained from token attributes are matched against groups in Keystone but the user is not included into those groups in Keystone DB (to support dynamic group membership changes at the IdP side). If any of the target groups do not exist in Keystone authentication fails;
4) A domain-scoped federated token is created (e.g. by Horizon) and then a project-scoped token is created using the previous token as the authentication method.

(4) is where the problem occurs.

[Environment]
Queens, 19.04 charms, token-provider='uuid' for charm-keystone.

openstack commands used to configure an IdP: https://paste.ubuntu.com/p/nj6MdQDKk2/

keystone.conf sections:
[auth]
methods = external,password,token,oauth1,totp,application_credential,saml2
[federation]
trusted_dashboard = https://dashboard.maas/auth/websso/
[saml2]
remote_id_attribute = MELLON_IDP

IdP is ADFS in this case which uses a windows account name as NAMEID and adds an attribute which corresponds to a group ID (the group name in Active Directory is the OpenStack group ID). The resulting SAML token then contains the following elements:
 <AttributeStatement> <Attribute Name="groups"> <AttributeValue>3f031869ef9f4dc49a342d6be69e98b3</AttributeValue> </Attribute> </AttributeStatement>

The direct usage of a group ID is present to rule out group name to ID resolution problems.

[Use-case]

Automatic project provisioning and Member role assignment to users is not used on purpose to manage user access to projects via group to role assignments. A user is a assigned to a group at the IdP side and the keystone database does not contain any role assignments for shadow-mapped users. `openstack role assignment list --names` will not contain anything related to group assignments - all group membership information will only be exposed in a token.

[Problem Description]

1) the first token (federated, obtained via v3 federation API) is domain-scoped and authentication succeeds for it;
2) then a client (e.g. Horizon) gets a project-scoped token based on that federated token (token authentication & regular v3 API) for which roles need to be populated - including the roles to access the target project;
3) the roles for the second token are not populated correctly based on the (dynamic) group assignments that came from the SAML token for the first token - clearly the role population code-path for the second token is not aware of groups that came dynamically with the SAML token. The expected result would be awareness of groups assigned to the shadow mapped user and then inference of roles from groups based on group to role assignments in the Keystone DB. This explains the fact that project auto-provisioning and project role assignment to shadow users directly works properly (because this can be queried by keystone from its db).

The visible end-result for a user authenticating via the dashboard is represented in a form of errors such as "Unauthorized: Unable to..." for any accessed dashboard pane.

[Symptoms]

Example: https://paste.ubuntu.com/p/syxxWmdyD7/
(keystone.token.provider): 2019-05-07 17:47:01,947 DEBUG Unable to validate token: The request you have made requires authentication.

Project-scoped token example (contains the right group and "methods": ["token", "saml2"]) as queried directly from the db:
https://paste.ubuntu.com/p/rRgXSctgWT/

rpdb trace - first pass at finding where it fails:
https://paste.ubuntu.com/p/DhG4HXCnBB/

Second pass (the most useful) - a trace point in keystone/token/providers/common.py get_token_data() going down to keystone/token/providers/common.py(432)_populate_roles() where the Unauthorized exception is thrown:
https://paste.ubuntu.com/p/pjRf7qBzcX/

[Root Cause]

Based on the symptoms it is clear that _populate_roles (unlike populate_roles_for_federated_user) does not include group roles for groups obtained via federated authentication:

https://opendev.org/openstack/keystone/src/branch/stable/queens/keystone/token/providers/common.py#L408-L432 (_populate_roles)

https://opendev.org/openstack/keystone/src/branch/stable/queens/keystone/token/providers/common.py#L168-L193 (_get_roles_for_user, has a branch to work with group roles but for system-scoped tokens only)

https://opendev.org/openstack/keystone/src/branch/stable/queens/keystone/token/providers/common.py#L190-L193 (get_roles_for_user_and_project gets user to role assignments which are not present in this case)

Which in the end leads to exception.Unauthorized being thrown by Keystone https://opendev.org/openstack/keystone/src/branch/stable/queens/keystone/token/providers/common.py#L416-L432

[Mitigation]

Usage of fernet tokens for which the same behavior does not occur.

For existing openstack-charm-deployed environments, running this on a live cloud is the way forward:
juju config keystone token-provider='fernet'

# see the upgrade notes https://specs.openstack.org/openstack/charm-specs/specs/rocky/implemented/keystone-fernet-tokens.html#upgrading-an-existing-system

Note: this is more of a documentation bug as UUID tokens were removed in Rocky and so there is no point in fixing this at the distro level or upstream.

[Dmitrii Shcherbakov (dmitriis)]

Added Keystone (upstream), however, I am expecting a "won't fix" since this is about UUID tokens and it works with fernet.

[Colleen Murphy (krinkle)]

Queens is still maintained upstream[1] so we would still be happy to accept patches for it as long as we can confirm the behavior.

[1] https://releases.openstack.org/index.html#release-series

[Colleen Murphy (krinkle)]

Marking this as low priority for keystone, we'd accept a patch if it conforms to the stable branch guidelines but I suspect fixing it may be too big of a change for such an old release.

Are there issues with migrating to fernet?

[Dmitrii Shcherbakov (dmitriis)]

Hi Colleen,

Thanks for the response.

Migrating to fernet was not a problem for my case and I think it is a logical thing to do considering UUID tokens were removed in Rocky.

I agree that fixing it in an older release is a more expensive operation than migration to fernet tokens so for my case it did not justify the effort to write a patch.

I think this bug could be left open as a "known issue" until somebody needs this for UUID tokens and has time to implement the patch or Queens goes EOL.