cannot use trusts with federated users

[György Szombathelyi]

Deploying with federated user throws an exception in murano-engine with:

Exception Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)

The mentioned role is _member_

The full trace:

2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/engine.py", line 159, in execute
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self._create_trust()
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/engine.py", line 282, in _create_trust
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self._session.token, self._session.project_id)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/auth_utils.py", line 98, in create_trust
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine project=project)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/contrib/trusts.py", line 75, in create
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return f(*args, **new_kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 339, in create
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self.key)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 171, in _post
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine resp, body = self.client.post(url, body=body, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 179, in post
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return self.request(url, 'POST', **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return self.session.request(url, method, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return func(*args, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 420, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine raise exceptions.from_response(resp, method, url)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine NotFound: Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404) (Request-ID: req-760d033b-e456-4915-b197-e450d4c8a405)

Seems something wrong with creating a trust.

[Kirill Zaitsev (kzaitsev)]

Can you please provide some more information on how to reproduce this bug?

[György Szombathelyi (gyurco)]

Just setup OIDC federation, and try to deploy any environment with a federated user. But seems this problem also exists with heat, when trusts are enabled, maybe its a more general problem (or I forgot something during OIDC setup, but everything else works perfectly).

Workaround for me now is:
murano:
[engine]
use_trusts = False

heat:
deferred_auth_method = password

[György Szombathelyi (gyurco)]

Forgot to mention: in the same openstack setup, Murano works perfectly with a normal (non-federated) keystone user.

[Nikolay Starodubtsev (starodubcevna)]

As far as I can send that this bug related to keystone? from discussion in ML

[Dolph Mathews (dolph)]

I imagine this will be addressed by (or nearly addressed by) having concrete role assignments for federated users in keystone: https://review.openstack.org/#/c/284943/

[Steve Martinelli (stevemar)]

Anyway we can get this checked again? the patch that dolph cited has been merged and I think that should resolve this bug.

[Kirill Zaitsev (kzaitsev)]

Is this still an issue?

[György Szombathelyi (gyurco)]

Sorry for not respondig before, now I had a chance to test with Newton-rc1, and the same issue is there:
Failed to find roles [u'_member_'] for user 6a299f31021c417bb15e0de347d0528

However I did not change the mapping setup, is there anything special needed to use the new shadow users feature?

[György Szombathelyi (gyurco)]

This is the log from keystone:

2016-10-03 11:57:51.459 4253 DEBUG keystone.policy.backends.rules [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] enforce identity:create_trust: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <KeystoneToken (audit_id=fAxTlUK5Q6G1gPi6nq4M0A, audit_chain_id=fAxTlUK5Q6G1gPi6nq4M0A) at 0x7fefc0721178>, 'group_ids': ['5d70f4ee389b4c77940ed68cce777778'], 'project_id': u'dc76f45b6c6e4b839e63c6a193d754b9', 'trust_id': None, 'project_domain_id': u'default'} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2016-10-03 11:57:51.460 4253 DEBUG keystone.common.controller [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:163
2016-10-03 11:57:51.507 4253 WARNING keystone.common.wsgi [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab

[György Szombathelyi (gyurco)]

Ok, progressed further:

Seems it still doesn't work with the current method of assigning the role through a pre-created group, and assigning this group to the federated users.
So did a direct role assignment to the federated users, since it is possible now.
However still there are some issues:

1. Because this user doesn't exist until she/he tries to log in, one cannot provision user roles beforehand.

2. Still there's an issue with the trust creation (logs from keystone follows):

2016-10-03 13:16:11.411 26415 DEBUG keystone.middleware.auth [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: auth_context: {'is_delegated_auth': False,
 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <Keysto
neToken (audit_id=ZZsqGTjjRACs_Hny2yXXIg, audit_chain_id=ZZsqGTjjRACs_Hny2yXXIg) at 0x7f6b37da0e30>, 'group_ids': [], 'project_id': u'cb6344592c4e4fb99a15ccef715ada25', 'trust_id': None, 'project_domain_id': u'default'} fill_context /usr/
lib/python2.7/dist-packages/keystone/middleware/auth.py:243
2016-10-03 13:16:11.414 26415 INFO keystone.common.wsgi [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] POST http://cloudtest.duodecadits.com:35357/v3/OS-TRU
ST/trusts
2016-10-03 13:16:11.415 26415 DEBUG keystone.common.controller [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: Authorizing identity:create_trust(trust=
{u'impersonation': True, u'project_id': u'cb6344592c4e4fb99a15ccef715ada25', u'trustor_user_id': u'6a299f31021c417bb15e0de347d0528f', u'roles': [{u'name': u'_member_'}, {u'name': u'_member_'}], u'trustee_user_id': u'1345a05dcf2a442592fb77
0d53c114a0'}) _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:80
2016-10-03 13:16:11.417 26415 DEBUG keystone.policy.backends.rules [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] enforce identity:create_trust: {'is_delega
ted_auth': False, 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None,
 'token': <KeystoneToken (audit_id=ZZsqGTjjRACs_Hny2yXXIg, audit_chain_id=ZZsqGTjjRACs_Hny2yXXIg) at 0x7f6b37da0e30>, 'group_ids': [], 'project_id': u'cb6344592c4e4fb99a15ccef715ada25', 'trust_id': None, 'project_domain_id': u'default'} e
nforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2016-10-03 13:16:11.418 26415 DEBUG keystone.common.controller [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: Authorization granted inner /usr/lib/pyt
hon2.7/dist-packages/keystone/common/controller.py:163
2016-10-...

[Robert Duncan (rduncan-t)]

Noticed this is marked incomplete, I have logged a similar (exactly the same) bug report based on SAML users.
https://bugs.launchpad.net/fuel/+bug/1626046

it is accepted and targeted for 9.2

[Lance Bragstad (lbragstad)]

In response to comment #10. Keystone has merged a spec that will allow deployers to define more descriptive mappings [0]. This should mitigate #1 from comment #10.

Since using a direct role assignment works around this bug, we can track the progress of the mapping engine with the blueprint [1].

[0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ocata/shadow-mapping.html
[1] https://blueprints.launchpad.net/keystone/+spec/shadow-mapping

[György Szombathelyi (gyurco)]

Still the same with keystone 10.0.0 final:

2016-10-25 10:00:55.205 11842 DEBUG keystone.middleware.auth [req-12241e76-3bc1-43de-8343-60c637e4a1ec 2168f89338624e2abe8f2aa30b415c90 aab8cd701f76426a9316e8715e2f6e7c - Federated default] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '2168f89338624e2abe8f2aa30b415c90', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <KeystoneToken (audit_id=lNaiJ_o3SOGBJ2JKL3gfTQ, audit_chain_id=lNaiJ_o3SOGBJ2JKL3gfTQ) at 0x7f7fc076d990>, 'group_ids': [], 'project_id': u'aab8cd701f76426a9316e8715e2f6e7c', 'trust_id': None, 'project_domain_id': u'default'} fill_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:243

The _member_ role is duplicated, so there'll be an error storing the trust:

2016-10-25 09:53:51.110 11838 WARNING keystone.common.wsgi [req-e57d996d-b5a7-428b-86b3-72e2d4c13250 2168f89338624e2abe8f2aa30b415c90 aab8cd701f76426a9316e8715e2f6e7c - Federated default] Conflict occurred attempting to store trust - Duplicate Entry

[Robert Duncan (rduncan-t)]

looks like something in the keystone policy is enumerating the role twice? - perhaps for user and group?

RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '2168f89338624e2abe8f2aa30b415c90', 'roles': [u'_member_', u'_member_'],

[Robert Duncan (rduncan-t)]

the policy was also patched in shadow-users bp

https://review.openstack.org/#/c/284943/63/etc/policy.v3cloudsample.json

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/415545

[Davanum Srinivas (DIMS) (dims-v)]

Boris will be raising awareness of this issue and review (https://review.openstack.org/#/c/415545/) at the keystone team meeting tomorrow

[Artem Andreev (just-wow)]

We're also observing this issue in a customer environment and managed to delve deeper into Keystone internals to identify the root cause. Results of the research please find below, probably it would be a good idea to update the bug description with these details. The issue has been confirmed to be present in all releases, including Ocata (not yet released as of now).

Pre-condition
=============
- An OpenStack cloud federated w/ a SAML2 Identity Provider
- Heat configured to use Keystone trusts to do deferred operations

Steps to Reproduce
==================
1. Create a mapping in order to allow federated users with "cloud_admin" and "project_admin" roles to access the cloud.
2. Login to Horizon using the IdP credentials.
3. Navigate to "Orchestration" tab, press "Create Stack" button.
4. Provide stack details and press "Create" button.

Expected Result
===============
Stack is created and activated.

Observed Result
===============
Stack creation fails. Horizon Web UI displays error message "Unknown role 'cloud_admin'" for cloud_admin user.

Root Cause Analysis
===================
Internal Workflow
-----------------
1. A federated user calls Heat API to create a new stack providing a project-scoped token
2. If "trusts" is configured as the preferred mechanism to handle deferred operations on the stack, Heat Engine tries to create a new trust for the user's context
3. Heat Engine calls Keystone API on behalf of the user to create a new trust object
4. Keystone checks if the trustor (the user) actually is actually associated with the requested roles in the context of the project
  a. To do that Keystone calls it's assignments subsystem to list the user's roles in the subject project
  b. Assignments subsystem by turn calls identity subsystem to list the users's groups in the subject project
  c. Configured identity backend (SQL) is not able to find a federated user in the DB and raises UserNotFound exception
5. Keystone API returns "404 user not found" error code to Heat
6. Heat misinterprets the "404 not found" response as Keystone's inability to find a requested role, stack creation fails

Summary
-------
The Federated users can't use cloud orchestration services that perform deferred actions on cloud on behalf of the user.
This happens because Heat tries to treat Federated user as regular Keystone user which usually has ID, assigned project(s) and assigned role(s).

However, federated users don't belong to Keystone database. They get mapped to user groups and inherit group roles and projects in runtime, during the authentication process. When someone, for example, Heat, tries to query Keystone about federated user, this will lead to Identity Error and, as a consequence, make Heat fail.

[Lance Bragstad (lbragstad)]

Automatically unassigning due to inactivity.

[Thomas Herve (therve)]

After chatting with Lance and the Keystone people at the PTG, they convinced me to try again and check if it worked.

The summary: it did for me.

I used the fairly new devstack plugin from Keystone that deploys with federation, backed by testshib.
I used the following map: http://paste.openstack.org/show/621059/ . I used to following script to test http://paste.openstack.org/show/621060/ (Thanks Colleen for all the pointers).

I made some back and forth, so I don't completely trust (ah) my setup, but I managed to create a stack with trust behind. I didn't get the issue mentioned by Artem where the user couldn't be found. With shadow mapping, the users do exist in the keystone database, and get roles assignment.

I used master, but the code ought to be in ocata, it would be worth checking if it works there. Tests with a real deployment would be great, too.

At that point, I think that bug could be closed by a documentation fix. We talked about an integration test in Keystone as well to make sure trusts work with federation (as there are federation tempest tests now).

[Lance Bragstad (lbragstad)]

Thanks for the follow up, Thomas. Are there any patches in review for documentation or testing? I want to make sure we include them here before closing this out.

[John Garbutt (johngarbutt)]

I am trying this using the map to a group (rather than auto create assignment on first login) and it seems to fail in the same way.

Now I need to use a group so the level of assurance (did I use two factor auth, is it just a random google account, etc) for the particular login ensures the user gets the correct permissions each time.

(I think this is using pike or ocata, I am just doing the double checking of that).

[John Garbutt (johngarbutt)]

Just to confirm, seeing this with pike and dynamic group mapping (i.e. no mapping to specific roles and projects, only map to groups)

[Steven Relf (srelf)]

This still seems to be an issue if you don't use concreate mappings.

For example if you have multiple customers with multiple projects a mapping file like the on eabove in comment#20 becomes unwildy.

[Colleen Murphy (krinkle)]

This should be possible with group membership mappings now that we've completed https://bugs.launchpad.net/keystone/+bug/1809116

Would someone in this thread be willing to try the feature out using master in keystone and give us feedback? https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html#create-a-mapping

[Steven Relf (srelf)]

i would love too. Ill see if i can get something setup

[Steven Relf (srelf)]

So, any ideas on the quickest way to get something stood up to test this, normally i would use packstack but there doesnt seem to be a u-release version as yet

[David Wilde (dave-wilde)]

Based on my testing this is now working in master due to the switch to Secure RBAC. Please let me know if this continues to be an issue and we can revisit with fresh logs.