Failed to create sahara cluster if reauthentication_auth_method =trusts

Bug #1717918 reported by Wang Bo on 2017-09-18
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Undecided
Unassigned
OpenStack Identity (keystone)
Low
Unassigned
Sahara
New
Undecided
Unassigned

Bug Description

set reauthentication_auth_method =trusts

heat stack-create sahara_cluster -f sahara_cluster.yaml

sahara-engine came into error when to get image, with following error:
Forbidden: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403)

The error log is very clear, heat passed trust scoped token to sahara.
Setting "reauthentication_auth_method =trusts" works well for other components but sahara.

sahara_cluster.yaml:
heat_template_version: 2015-04-30

resources:
  sahara_cluster:
    type: OS::Sahara::Cluster
    properties:
      name: rdo
      plugin_name: spark
      hadoop_version: 1.6.0
      default_image_id: 4ae3ae1a-ca7f-428b-ae2b-782fb81a9dd2
      cluster_template_id: 7c8f8453-0934-4270-917f-7e3188320a9a
      key_name: stack
      neutron_management_network: private
      use_autoconfig: true

Wang Bo (chestack) on 2017-09-18
affects: heat → sahara
affects: sahara → heat
Luigi Toscano (ltoscano) wrote :

Is it the last sahara master? Is it pike? What is the content of sahara.conf? Can you please share some details from your deployment?

Shu Yingya (felixshu) wrote :

Luigi, this problem affects us from mitaka to stable/pike at least. I don't have master branch to test it right now.
This env is latest RDO deployed by packstack. and the sahara.conf is the default value without any change.

Rabi Mishra (rabi) wrote :

AFAICT, from heat point of view it should work from newton onwards after https://github.com/openstack/heat/commit/cd125f328e3e4e9f517278b04347515eba380d5a#diff-661fd41a079826552bb7477bd817ea8f landed.

Are you sure you get the above error in pike?

Shu Yingya (felixshu) wrote :

Hi Rabi, I have retried it on my env (pike RDO) and it can be reproduced 100%.

Xiaoyang Zhang (es-xiaoyang) wrote :

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+ trust_token +--------+ heat_trust_token +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ 403 error +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :
                                                          heat_trust_token
                         reauthentication_auth_method =trusts +
             user_token +------+ trust_token +--------+ auth_ref +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ complete +----------------+
                                                    \
                                                     \ create_trust
                                            403 error \ +----------+
                                                       -> | keystone |
                                                          +----------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

Xiaoyang Zhang (es-xiaoyang) wrote :

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+
                        | sahara |
                        +--------+
                            |
                 403 error | heat_trust_token
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+ create_trust +----------+
                        | sahara | ------------> | keystone |
                        +--------+ 403 error +----------+
                            |
                            | heat_trust_token
                   complete | +
                            | auth_ref
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

Rabi Mishra (rabi) wrote :

> We also need to modify heat to create trust. Add allow_redelegation=True parameter

You can set allow_redelegation=True in [trust] section of keystone.conf[1.]

However, redelegation does not seem to work atm (possibly keystone issue) as discussed in[2].

[1] https://github.com/openstack/keystone/blob/fab399e26cdbe7cffba895f99d7247896ec6cb82/keystone/conf/trust.py#L25

[2] https://bugs.launchpad.net/heat/+bug/1701498

Wang Bo (chestack) wrote :

Hi Rabi, AFAIK reauthentication_auth_method =trusts was introduced in[1] to defeat token expiry by reauthentication. So if set reauthentication_auth_method = "", the long-running stack creation will be failed by token expiry? Could you help explain it?

[1] https://review.openstack.org/#/c/226384/

Rabi Mishra (rabi) wrote :

Yeah, that's in heat (we use trusts auth plugin with keystone session which would request for another token if the token expires(401][1]. But if sahara uses heat provided trust token to again request for a token, then it would fail.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726

Wang Bo (chestack) wrote :

Rabi, thanks for your comment.
From the link you pasted[1], The re-authentication happened in keystone that looks like working for all kinds keystone session(not only trusts auth plugin but also [2]). If that we could leave reauthentication_auth_method = ""(re-authentication will still work). Please correct me if anything wrong.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726
[2] https://github.com/openstack/heat/blob/master/heat/common/context.py#L283-L284

Rabi Mishra (rabi) wrote :

No it would not work as by default we use the access plugin[1] with the token_info we get from the auth middleware.

The reauth would work only cases where password and trust plugins are used.

[1] https://github.com/openstack/heat/blob/master/heat/common/context.py#L251

Lance Bragstad (lbragstad) wrote :

This might be something we can improve in the keystone documentation. It appears keystone has some limitations on what you can do with impersonation and redelegation [0].

Unfortunately, I can't seem to find clear documentation on it. I'll dig a bit more, but this might classify as a documentation fix for keystone.

[0] https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_trust.py#L383-L386

tags: added: office-hours
Changed in keystone:
status: New → Triaged
importance: Undecided → Low
tags: added: documentation
Rico Lin (rico-lin) on 2018-05-07
Changed in heat:
milestone: none → no-priority-tag-bugs
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers