federated users cannot use Murano or Sahara

[Robert Duncan]

Users considering using that feature, please read comment #17 first https://bugs.launchpad.net/fuel/+bug/1626046/comments/17

fuel fuel-version
api: '1'
auth_required: true
feature_groups:
- experimental
- advanced
openstack_version: mitaka-9.0
release: '9.0'

Description:
With keystone configured for federated access (SAML) Sahara cannot create trusts due to 'unable to find role'

Steps to reproduce:
Configure keystone for federation such as SAML or OID

Expected Results:
federated users are ephemeral but consume a role (_member_) based on group membership, role based access should work.

actual results:
Sahara - RESP BODY: {"error": {"message": "Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab", "code": 404, "title": "Not Found"}}

and finally:
Unable to create trust (reason: Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404)

 openstack role list (doesn't seem to make much sense Sahara can't find the _member_ role)
+----------------------------------+-----------------+
| ID | Name |
+----------------------------------+-----------------+
| 79fedf162a664cdbb6de7117d4998566 | ResellerAdmin |
| 87cf7f569672416db4027839aaa93eec | heat_stack_user |
| 897d116732174ee8be888aba12b7a550 | admin |
| 9fe2ff9ee4384b1894a90878d3e92bab | _member_ |
+----------------------------------+-----------------+

this goes for Murano and heat also - the same result - the federated users role cannot be found.

Sahara snippet:

 DEBUG keystoneclient.auth.identity.v3.base [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Making authentication request to http://172.25.60.5:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:188
2016-09-20 17:02:47.302 27869 DEBUG keystoneclient.auth.identity.v3.base [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Making authentication request to http://172.25.60.5:5000/v3/auth/tokens get_auth_ref /usr/lib/python2.7/dist-packages/keystoneclient/auth/identity/v3/base.py:188
2016-09-20 17:02:47.412 27869 DEBUG keystoneclient.session [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] REQ: curl -g -i --insecure -X POST http://172.25.60.5:35357/v3/OS-TRUST/trusts -H "User-Agent: python-keystoneclient" -H "Content-Type: application/json" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}cbd44946362e5abd7f0a4bd25512a85a178cc87c" -d '{"trust": {"impersonation": true, "trustor_user_id": "43951d0af69848268f07937c12fa36fd", "allow_redelegation": true, "roles": [{"name": "_member_"}], "trustee_user_id": "d29432ca2fdf4b0996b01347155934df", "project_id": "307a375b43274f779d89b0512824a054"}}' _http_log_request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:206
2016-09-20 17:02:47.560 27869 DEBUG keystoneclient.session [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] RESP: [404] Content-Length: 114 Vary: X-Auth-Token Server: Apache Connection: close Date: Tue, 20 Sep 2016 17:02:47 GMT Content-Type: application/json x-openstack-request-id: req-b74e985d-d32d-4857-8f3c-ec4c014fb107
RESP BODY: {"error": {"message": "Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab", "code": 404, "title": "Not Found"}}
 _http_log_response /usr/lib/python2.7/dist-packages/keystoneclient/session.py:231
2016-09-20 17:02:47.561 27869 DEBUG keystoneclient.session [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Request returned failure status: 404 request /usr/lib/python2.7/dist-packages/keystoneclient/session.py:419
2016-09-20 17:02:47.562 27869 ERROR sahara.service.trusts [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Unable to create trust (reason: Could not find role: 9fe2ff9ee4384b1894a90878d3e92bab (HTTP 404) (Request-ID: req-b74e985d-d32d-4857-8f3c-ec4c014fb107))
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Error during rollback of cluster (reason: Failed to create trust
Error ID: 2d5ce02c-0fb6-4856-9d29-d0f085c0e993)
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Traceback (most recent call last):
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] File "/usr/lib/python2.7/dist-packages/sahara/service/ops.py", line 210, in wrapper
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] if _rollback_cluster(cluster, ex):
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] File "/usr/lib/python2.7/dist-packages/sahara/service/ops.py", line 238, in _rollback_cluster
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] _setup_trust_for_cluster(cluster)
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] File "/usr/lib/python2.7/dist-packages/sahara/service/ops.py", line 180, in _setup_trust_for_cluster
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] trusts.create_trust_for_cluster(cluster)
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] File "/usr/lib/python2.7/dist-packages/sahara/service/trusts.py", line 97, in create_trust_for_cluster
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] allow_redelegation=True)
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] File "/usr/lib/python2.7/dist-packages/sahara/service/trusts.py", line 75, in create_trust
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] raise ex.CreationFailed(_('Failed to create trust'))
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] CreationFailed: Failed to create trust
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Error ID: 2d5ce02c-0fb6-4856-9d29-d0f085c0e993
2016-09-20 17:02:47.698 27869 ERROR sahara.service.ops [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3]
2016-09-20 17:02:48.182 27869 INFO sahara.utils.cluster [req-3b19bf3e-6bbb-4e6f-88cb-921a3cf8bbca 43951d0af69848268f07937c12fa36fd 307a375b43274f779d89b0512824a054 - - -] [instance: none, cluster: 58f9e615-9ac1-4445-8c01-ac2d766b02c3] Cluster status has been changed. New status=Error

Reproducibility:
 configure keystone v3 API for federation

Workaround:
 unknown - I would especially like to know of any workaround this? any feedback is appreciated

Impact:
 federation users cannot access Sahara and Murano or heat, this would mean, for me, needing to create manual accounts for +500 students and revert to keystone v2 with users in SQL backend, breaks SSO, identity lifecycle management and UX.

Description of the environment:
 Operation system: Ubuntu with MOS packages
 Versions of components: fuel 9.0
 Reference architecture: HA with ceph
 Network model: GRE
 Related projects installed: Murano, Sahara
Additional information:
 here's something weird - EVERYTHING else works with federated users including swift with ceph hammer backend, but apparently this should not work until ceph jewel
https://bugs.launchpad.net/mos/10.0.x/+bug/1498552

How are the other services working with role based access as group members? - they are not relying yet on the shadow users bp, can the same approach be configured in murano, sahara, heat?

this bug is addressed here:
https://review.openstack.org/#/c/284943/

can this be backported to mitaka? or can I patch keystone manually in the meantime.

[Robert Duncan (rduncan-t)]

In fact this is a heat issue, heat cannot create the keystone trust because it can't find the federated users role assignment.

[Robert Duncan (rduncan-t)]

Oleksii, if you need any more information please let me know

[Dmitry Pyzhov (dpyzhov)]

Looks like Oleksii doesn't work with the issue. Moving back to the team.

[Nikolay Pliashechnikov (npliashechnikov)]

Hello everyone,

I have recently investigated deeper into this issue.
Shortly, there is a way to use Heat and its dependant services on MOS version 9, though it requires significant changes in the Keystone and, probably, authorization model.

The core reason behind this issue is that Heat tries to use Keystone V3 Trusts API in order to be able to perform deferred actions on behalf of the user who created stack, such as scale up/scale down. There is no way to fix this in Heat, as all logic relies on this functionality (and works perfectly with SQL and LDAP users), so the fixes should be made on Keystone's side.

When Keystone issues scoped token for a federated user, it uses User ID supplied by Identity Provider. Keystone also stores shadow copy of federated user in its DB on a first login. However, in this case, it DOES NOT use the ID supplied by IdP - instead of this, it generates its own uuid4() ID!

So, when Heat goes to Keystone and asks to create a trust for the user_id (IdP-supplied) specified in the token, Keystone can't find anything similar in its DB as IDs do not match. And, if you can make Keystone use User IDs provided by IdP to create shadow users, half of the problems are solved.

Another issue is with the group-based access model that is currently used with federated authentication. After some testing I have found that user with group-derived roles is not able to grant trusts, as Keystone does not take it into account when validating if the user is eligible to create trust with given permissions.
There is currently no fix for that behaviour that I am aware of, but, since Newton release, individual role assignment are supported for federated users. This was made possible by this change:
https://review.openstack.org/#/c/284943/
I believe it can be backported to Keystone Mitaka. This, however, would probably require some changes in the users' mapping rules.

I was able to use Heat on MOS 9.0 after explicitly assigning admin role to a federated user. Before that I made sure that ID of the user in Keystone DB is the same as supplied by IdP.

I'm adding patch that forces Keystone to take IdP-supplied user ID to its DB.

[Robert Duncan (rduncan-t)]

this is great news! - thanks Nikolay. - there is one thing I have noticed in the shadow users blue print and that is the change to keystone policy file, I don't know if this will have any affect on ...

"Another issue is with the group-based access model that is currently used with federated authentication. After some testing I have found that user with group-derived roles is not able to grant trusts, as Keystone does not take it into account when validating if the user is eligible to create trust with given permissions."

...but the change by keystone policy.json

this was added:

"identity:list_projects_for_user": "",
"identity:list_domains_for_user": "",

and this was removed:

"identity:list_projects_for_groups": "",
"identity:list_domains_for_groups": "",

it's not obvious to me why this is a requirement for federated access - the upgrade note only states:

upgrade:
  - In the policy.json file, we changed `identity:list_projects_for_groups`
    to `identity:list_projects_for_user`. Likewise, we changed
    `identity:list_domains_for_groups` to `identity:list_domains_for_user`. If
    you have customized the policy.json file, you will need to make these
    changes. This was done to better support new features around federation.

this sort of goes against operator intuition as one would expect federated 'users' to be ephemeral and consume their groups role - now it seems that federated users will also be keystone users which is not usual. (e.g. AWS federated users are not IAM users - simple RBAC - also the mapping is not nearly as awful especially for a single user with multiple roles)

[Vitaly Sedelnik (vsedelnik)]

Related bugs - https://bugs.launchpad.net/fuel/+bug/1592758 and https://bugs.launchpad.net/keystone/+bug/1589993. The latter has a fix on review, let's wait for upstream fix to be merged and see if it helps. Retargeting to 9.3.

[Peter Razumovsky (prazumovsky)]

After dicussion with Nikolay, we concluded that there is no issue in Heat - it's Keystone side issue. So, I remove assignee and ask keystone team to assign on this bug.

[Davanum Srinivas (DIMS) (dims-v)]

Please follow the upstream bug for updates

[Davanum Srinivas (DIMS) (dims-v)]

Update from Boris:

Trusts with federation don't really work together, and in fact are not
supposed to. The fact that we didn't explicitely say that in keystone is
a huge mistake.

They don't work because we cannot know what happens on the remote side,
in IdP. For example, if user gets moved from employees to ex-employees,
there is no way for us to notices that. The following situation is possible:

Let user A be in group "employees". Being one of employees, the user
creates a trust for user B. Then A gets dismissed, removed from group
employees and added to group ex-employees. Keystone doesn't know about
that. User A can now authenticate as user B and restore all roles they
previously had.

The solution for that could be limiting trust lifetime, as was suggested
at the meeting yesterday. It would work for Murano. But doen't work for
Heat. Heat uses trusts to perform operations long after the token has
expired. For example, autoscaling can get triggered in 30 days after it
gets created. And at that moment Heat requires the trust.

[Robert Duncan (rduncan-t)]

Using the above logic this would mean the same thing for ldap users? unless trusts are somehow conditional, is keystone looking back into the ldap directory to see if the trustor is active/enabled?

[Denis Meltsaykin (dmeltsaykin)]

I'm going to re-target this to Fuel 11 milestone, since there is no visible progress on the issue. For that reason also moving from 9.2-mu-1.

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/1.0/mitaka
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/32508

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/29993
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 95d22473e5042b373c4f0c675a242bd42d148631
Author: Boris Bobrov <email address hidden>
Date: Mon Apr 3 07:32:40 2017

Enable trusts for federated users

1. Add federated shadow user to groups on user creation and remove from
old groups
2. Do not always check domains because federated user doesn't belong
to a domain

Closes-Bug: #1626046
Closes-Bug: https://mirantis.jira.com/browse/PROD-9948

Change-Id: I2dd601b7bdc56f2c19628304095d0f7be0a33d87

[Michael Semenov (msemenov)]

Verified both on the QA environment and the customer's lab.

[Robert Duncan (rduncan-t)]

Nice! - but how come anonymous users can't view the fixes anymore?
https://review.fuel-infra.org/32508

[Dmitry Mescheryakov (dmitrymex)]

Robert, MOS OpenStack source repositories were always closed for people outside of Mirantis. On the other hand, OpenStack packages (specs) and dependencies repos are open. E.g.
https://review.fuel-infra.org/#/admin/projects/openstack-build/keystone-build
https://review.fuel-infra.org/#/admin/projects/packages/trusty/rabbitmq-server

But you may consume the resulting packages. If I am not mistaken, 9.3.0-3~u14.04+mos7 packages from the following mirror should contain the fix http://mirror.fuel-infra.org/mos-repos/ubuntu/snapshots/9.0-2017-04-03-122420/pool/main/k/keystone/

Please note that the behaviour is not enabled by default, as enabling it opens a security hole. To enable it, you need to set [federation]/cache_group_membership_in_db=True in /etc/keystone/keystone.conf and restart the apache2 service. I will write a standalone comment describing the vulnerability and how to mitigate it.

[Dmitry Mescheryakov (dmitrymex)]

To enable trusts for federated environment, users need to manually set the following parameter in /etc/keystone/keystone.conf:
[federation]/cache_group_membership_in_db=True

When turned on, Keystone will save in DB group membership for federated users. Enabling this option is the only way so far to make trusts delegated by federated users work.

The downside is that a trust could be used even after the delegating user's permissions were changed in IdP. For example, consider the following scenario:

1. User A has role admin in tenant T.
2. User A generates trust for user B, granting him that role.
3. Cloud admin removes the role from user A in Identity Provider (IdP).
4. But Keystone has cached the role assignment and still thinks that A owns admin role in tenant T.
5. As a result, user B still can use the trust and act as an admin in T.

That situation will be resolved only once user A logs into Keystone (that is the only moment when Keystone gets update about user from IdP).

The issue could be mitigated by an automation script which propagates changes in group assignments in IdP to the Keystone.

[Fuel Devops McRobotson (fuel-devops-robot)]

Reviewed: https://review.fuel-infra.org/32508
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/1.0/mitaka

Commit: 2239c530c2b63effb1cd7b9b8396826f22cfb20f
Author: Boris Bobrov <email address hidden>
Date: Thu Apr 6 10:05:58 2017

Enable trusts for federated users

1. Add federated shadow user to groups on user creation and remove from
old groups
2. Do not always check domains because federated user doesn't belong
to a domain

Closes-Bug: #1626046
Closes-Bug: https://mirantis.jira.com/browse/PROD-9948

Change-Id: I2dd601b7bdc56f2c19628304095d0f7be0a33d87

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/newton
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/33506

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: 11.0/ocata
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/33903

[Fuel Devops McRobotson (fuel-devops-robot)]

Fix proposed to branch: mcp/ocata
Change author: Boris Bobrov <email address hidden>
Review: https://review.fuel-infra.org/34598

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Roman Podoliaka <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33903
Reason: we don't use 11.0/ocata anymore - mcp/ocata is the correct branch name

[Alexey Stupnikov (astupnikov)]

Won't fix for 8.0-updates as it is a feature.

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33506
Reason: already in mcp/ocata branch,
https://review.fuel-infra.org/gitweb?p=openstack/keystone.git;a=commit;h=98bc10e930fd65ecce2be785829405e41ab0cf76

[Fuel Devops McRobotson (fuel-devops-robot)]

Change abandoned by Pavlo Shchelokovskyy <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34598
Reason: already have this in mcp/ocata
https://review.fuel-infra.org/gitweb?p=openstack/keystone.git;a=commit;h=98bc10e930fd65ecce2be785829405e41ab0cf76