cannot use trusts with federated users

Bug #1589993 reported by György Szombathelyi on 2016-06-07
64
This bug affects 12 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
High
Unassigned

Bug Description

Deploying with federated user throws an exception in murano-engine with:

Exception Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)

The mentioned role is _member_

The full trace:

2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/engine.py", line 159, in execute
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self._create_trust()
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/engine.py", line 282, in _create_trust
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self._session.token, self._session.project_id)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/murano/common/auth_utils.py", line 98, in create_trust
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine project=project)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/v3/contrib/trusts.py", line 75, in create
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 75, in func
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return f(*args, **new_kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 339, in create
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine self.key)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/base.py", line 171, in _post
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine resp, body = self.client.post(url, body=body, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 179, in post
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return self.request(url, 'POST', **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 331, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 98, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return self.session.request(url, method, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/positional/__init__.py", line 94, in inner
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine return func(*args, **kwargs)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine File "/usr/lib/python2.7/dist-packages/keystoneclient/session.py", line 420, in request
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine raise exceptions.from_response(resp, method, url)
2016-06-07 15:08:05.732 8194 ERROR murano.common.engine NotFound: Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404) (Request-ID: req-760d033b-e456-4915-b197-e450d4c8a405)

Seems something wrong with creating a trust.

Kirill Zaitsev (kzaitsev) wrote :

Can you please provide some more information on how to reproduce this bug?

Changed in murano:
milestone: none → newton-2
György Szombathelyi (gyurco) wrote :

Just setup OIDC federation, and try to deploy any environment with a federated user. But seems this problem also exists with heat, when trusts are enabled, maybe its a more general problem (or I forgot something during OIDC setup, but everything else works perfectly).

Workaround for me now is:
murano:
[engine]
use_trusts = False

heat:
deferred_auth_method = password

György Szombathelyi (gyurco) wrote :

Forgot to mention: in the same openstack setup, Murano works perfectly with a normal (non-federated) keystone user.

As far as I can send that this bug related to keystone? from discussion in ML

Dolph Mathews (dolph) wrote :

I imagine this will be addressed by (or nearly addressed by) having concrete role assignments for federated users in keystone: https://review.openstack.org/#/c/284943/

Changed in keystone:
assignee: nobody → Ron De Rose (ronald-de-rose)
importance: Undecided → Wishlist
status: New → Triaged
Changed in murano:
importance: Undecided → Wishlist
Changed in murano:
milestone: newton-2 → newton-3
Steve Martinelli (stevemar) wrote :

Anyway we can get this checked again? the patch that dolph cited has been merged and I think that should resolve this bug.

Changed in keystone:
status: Triaged → Incomplete
assignee: Ron De Rose (ronald-de-rose) → nobody
Kirill Zaitsev (kzaitsev) wrote :

Is this still an issue?

Changed in murano:
status: New → Incomplete
Changed in murano:
milestone: newton-3 → newton-rc1
Changed in murano:
milestone: newton-rc1 → newton-rc2
Changed in murano:
milestone: newton-rc2 → ocata-1
György Szombathelyi (gyurco) wrote :

Sorry for not respondig before, now I had a chance to test with Newton-rc1, and the same issue is there:
Failed to find roles [u'_member_'] for user 6a299f31021c417bb15e0de347d0528

However I did not change the mapping setup, is there anything special needed to use the new shadow users feature?

György Szombathelyi (gyurco) wrote :

This is the log from keystone:

2016-10-03 11:57:51.459 4253 DEBUG keystone.policy.backends.rules [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] enforce identity:create_trust: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <KeystoneToken (audit_id=fAxTlUK5Q6G1gPi6nq4M0A, audit_chain_id=fAxTlUK5Q6G1gPi6nq4M0A) at 0x7fefc0721178>, 'group_ids': ['5d70f4ee389b4c77940ed68cce777778'], 'project_id': u'dc76f45b6c6e4b839e63c6a193d754b9', 'trust_id': None, 'project_domain_id': u'default'} enforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2016-10-03 11:57:51.460 4253 DEBUG keystone.common.controller [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] RBAC: Authorization granted inner /usr/lib/python2.7/dist-packages/keystone/common/controller.py:163
2016-10-03 11:57:51.507 4253 WARNING keystone.common.wsgi [req-7e3a3679-b2f0-4a47-9d11-7abd9edacd0a 6a299f31021c417bb15e0de347d0528f dc76f45b6c6e4b839e63c6a193d754b9 - Federated default] Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab

György Szombathelyi (gyurco) wrote :
Download full text (4.2 KiB)

Ok, progressed further:

Seems it still doesn't work with the current method of assigning the role through a pre-created group, and assigning this group to the federated users.
So did a direct role assignment to the federated users, since it is possible now.
However still there are some issues:

1. Because this user doesn't exist until she/he tries to log in, one cannot provision user roles beforehand.

2. Still there's an issue with the trust creation (logs from keystone follows):

2016-10-03 13:16:11.411 26415 DEBUG keystone.middleware.auth [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: auth_context: {'is_delegated_auth': False,
 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <Keysto
neToken (audit_id=ZZsqGTjjRACs_Hny2yXXIg, audit_chain_id=ZZsqGTjjRACs_Hny2yXXIg) at 0x7f6b37da0e30>, 'group_ids': [], 'project_id': u'cb6344592c4e4fb99a15ccef715ada25', 'trust_id': None, 'project_domain_id': u'default'} fill_context /usr/
lib/python2.7/dist-packages/keystone/middleware/auth.py:243
2016-10-03 13:16:11.414 26415 INFO keystone.common.wsgi [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] POST http://cloudtest.duodecadits.com:35357/v3/OS-TRU
ST/trusts
2016-10-03 13:16:11.415 26415 DEBUG keystone.common.controller [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: Authorizing identity:create_trust(trust=
{u'impersonation': True, u'project_id': u'cb6344592c4e4fb99a15ccef715ada25', u'trustor_user_id': u'6a299f31021c417bb15e0de347d0528f', u'roles': [{u'name': u'_member_'}, {u'name': u'_member_'}], u'trustee_user_id': u'1345a05dcf2a442592fb77
0d53c114a0'}) _build_policy_check_credentials /usr/lib/python2.7/dist-packages/keystone/common/controller.py:80
2016-10-03 13:16:11.417 26415 DEBUG keystone.policy.backends.rules [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] enforce identity:create_trust: {'is_delega
ted_auth': False, 'access_token_id': None, 'user_id': '6a299f31021c417bb15e0de347d0528f', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None,
 'token': <KeystoneToken (audit_id=ZZsqGTjjRACs_Hny2yXXIg, audit_chain_id=ZZsqGTjjRACs_Hny2yXXIg) at 0x7f6b37da0e30>, 'group_ids': [], 'project_id': u'cb6344592c4e4fb99a15ccef715ada25', 'trust_id': None, 'project_domain_id': u'default'} e
nforce /usr/lib/python2.7/dist-packages/keystone/policy/backends/rules.py:76
2016-10-03 13:16:11.418 26415 DEBUG keystone.common.controller [req-5f45f373-0097-4325-87f2-7dca317b71b8 6a299f31021c417bb15e0de347d0528f cb6344592c4e4fb99a15ccef715ada25 - Federated default] RBAC: Authorization granted inner /usr/lib/pyt
hon2.7/dist-packages/keystone/common/controller.py:163
2016-10-...

Read more...

Robert Duncan (rduncan-t) wrote :

Noticed this is marked incomplete, I have logged a similar (exactly the same) bug report based on SAML users.
https://bugs.launchpad.net/fuel/+bug/1626046

it is accepted and targeted for 9.2

Lance Bragstad (lbragstad) wrote :

In response to comment #10. Keystone has merged a spec that will allow deployers to define more descriptive mappings [0]. This should mitigate #1 from comment #10.

Since using a direct role assignment works around this bug, we can track the progress of the mapping engine with the blueprint [1].

[0] http://specs.openstack.org/openstack/keystone-specs/specs/keystone/ocata/shadow-mapping.html
[1] https://blueprints.launchpad.net/keystone/+spec/shadow-mapping

György Szombathelyi (gyurco) wrote :

Still the same with keystone 10.0.0 final:

2016-10-25 10:00:55.205 11842 DEBUG keystone.middleware.auth [req-12241e76-3bc1-43de-8343-60c637e4a1ec 2168f89338624e2abe8f2aa30b415c90 aab8cd701f76426a9316e8715e2f6e7c - Federated default] RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '2168f89338624e2abe8f2aa30b415c90', 'roles': [u'_member_', u'_member_'], 'user_domain_id': 'Federated', 'consumer_id': None, 'trustee_id': None, 'is_domain': False, 'trustor_id': None, 'token': <KeystoneToken (audit_id=lNaiJ_o3SOGBJ2JKL3gfTQ, audit_chain_id=lNaiJ_o3SOGBJ2JKL3gfTQ) at 0x7f7fc076d990>, 'group_ids': [], 'project_id': u'aab8cd701f76426a9316e8715e2f6e7c', 'trust_id': None, 'project_domain_id': u'default'} fill_context /usr/lib/python2.7/dist-packages/keystone/middleware/auth.py:243

The _member_ role is duplicated, so there'll be an error storing the trust:

2016-10-25 09:53:51.110 11838 WARNING keystone.common.wsgi [req-e57d996d-b5a7-428b-86b3-72e2d4c13250 2168f89338624e2abe8f2aa30b415c90 aab8cd701f76426a9316e8715e2f6e7c - Federated default] Conflict occurred attempting to store trust - Duplicate Entry

Robert Duncan (rduncan-t) wrote :

looks like something in the keystone policy is enumerating the role twice? - perhaps for user and group?

RBAC: auth_context: {'is_delegated_auth': False, 'access_token_id': None, 'user_id': '2168f89338624e2abe8f2aa30b415c90', 'roles': [u'_member_', u'_member_'],

Robert Duncan (rduncan-t) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/415545

Changed in keystone:
assignee: nobody → Boris Bobrov (bbobrov)
status: Incomplete → In Progress
no longer affects: murano

Boris will be raising awareness of this issue and review (https://review.openstack.org/#/c/415545/) at the keystone team meeting tomorrow

Changed in keystone:
assignee: Boris Bobrov (bbobrov) → Steve Martinelli (stevemar)
Changed in keystone:
milestone: none → ocata-3
assignee: Steve Martinelli (stevemar) → Boris Bobrov (bbobrov)
Changed in keystone:
milestone: ocata-3 → ocata-rc1
importance: Wishlist → High
summary: - Murano cannot deploy with federated user
+ cannot use trusts with federated users
Changed in keystone:
milestone: ocata-rc1 → next
Artem Andreev (just-wow) wrote :

We're also observing this issue in a customer environment and managed to delve deeper into Keystone internals to identify the root cause. Results of the research please find below, probably it would be a good idea to update the bug description with these details. The issue has been confirmed to be present in all releases, including Ocata (not yet released as of now).

Pre-condition
=============
- An OpenStack cloud federated w/ a SAML2 Identity Provider
- Heat configured to use Keystone trusts to do deferred operations

Steps to Reproduce
==================
1. Create a mapping in order to allow federated users with "cloud_admin" and "project_admin" roles to access the cloud.
2. Login to Horizon using the IdP credentials.
3. Navigate to "Orchestration" tab, press "Create Stack" button.
4. Provide stack details and press "Create" button.

Expected Result
===============
Stack is created and activated.

Observed Result
===============
Stack creation fails. Horizon Web UI displays error message "Unknown role 'cloud_admin'" for cloud_admin user.

Root Cause Analysis
===================
Internal Workflow
-----------------
1. A federated user calls Heat API to create a new stack providing a project-scoped token
2. If "trusts" is configured as the preferred mechanism to handle deferred operations on the stack, Heat Engine tries to create a new trust for the user's context
3. Heat Engine calls Keystone API on behalf of the user to create a new trust object
4. Keystone checks if the trustor (the user) actually is actually associated with the requested roles in the context of the project
  a. To do that Keystone calls it's assignments subsystem to list the user's roles in the subject project
  b. Assignments subsystem by turn calls identity subsystem to list the users's groups in the subject project
  c. Configured identity backend (SQL) is not able to find a federated user in the DB and raises UserNotFound exception
5. Keystone API returns "404 user not found" error code to Heat
6. Heat misinterprets the "404 not found" response as Keystone's inability to find a requested role, stack creation fails

Summary
-------
The Federated users can't use cloud orchestration services that perform deferred actions on cloud on behalf of the user.
This happens because Heat tries to treat Federated user as regular Keystone user which usually has ID, assigned project(s) and assigned role(s).

However, federated users don't belong to Keystone database. They get mapped to user groups and inherit group roles and projects in runtime, during the authentication process. When someone, for example, Heat, tries to query Keystone about federated user, this will lead to Identity Error and, as a consequence, make Heat fail.

Changed in keystone:
milestone: next → none
Lance Bragstad (lbragstad) wrote :

Automatically unassigning due to inactivity.

Changed in keystone:
assignee: Boris Bobrov (bbobrov) → nobody
status: In Progress → Triaged
Thomas Herve (therve) wrote :

After chatting with Lance and the Keystone people at the PTG, they convinced me to try again and check if it worked.

The summary: it did for me.

I used the fairly new devstack plugin from Keystone that deploys with federation, backed by testshib.
I used the following map: http://paste.openstack.org/show/621059/ . I used to following script to test http://paste.openstack.org/show/621060/ (Thanks Colleen for all the pointers).

I made some back and forth, so I don't completely trust (ah) my setup, but I managed to create a stack with trust behind. I didn't get the issue mentioned by Artem where the user couldn't be found. With shadow mapping, the users do exist in the keystone database, and get roles assignment.

I used master, but the code ought to be in ocata, it would be worth checking if it works there. Tests with a real deployment would be great, too.

At that point, I think that bug could be closed by a documentation fix. We talked about an integration test in Keystone as well to make sure trusts work with federation (as there are federation tempest tests now).

Lance Bragstad (lbragstad) wrote :

Thanks for the follow up, Thomas. Are there any patches in review for documentation or testing? I want to make sure we include them here before closing this out.

John Garbutt (johngarbutt) wrote :

I am trying this using the map to a group (rather than auto create assignment on first login) and it seems to fail in the same way.

Now I need to use a group so the level of assurance (did I use two factor auth, is it just a random google account, etc) for the particular login ensures the user gets the correct permissions each time.

(I think this is using pike or ocata, I am just doing the double checking of that).

John Garbutt (johngarbutt) wrote :

Just to confirm, seeing this with pike and dynamic group mapping (i.e. no mapping to specific roles and projects, only map to groups)

tags: added: federation
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers