Failed to create sahara cluster if reauthentication_auth_method =trusts

[Wang Bo]

set reauthentication_auth_method =trusts

heat stack-create sahara_cluster -f sahara_cluster.yaml

sahara-engine came into error when to get image, with following error:
Forbidden: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403)

The error log is very clear, heat passed trust scoped token to sahara.
Setting "reauthentication_auth_method =trusts" works well for other components but sahara.

sahara_cluster.yaml:
heat_template_version: 2015-04-30

resources:
  sahara_cluster:
    type: OS::Sahara::Cluster
    properties:
      name: rdo
      plugin_name: spark
      hadoop_version: 1.6.0
      default_image_id: 4ae3ae1a-ca7f-428b-ae2b-782fb81a9dd2
      cluster_template_id: 7c8f8453-0934-4270-917f-7e3188320a9a
      key_name: stack
      neutron_management_network: private
      use_autoconfig: true

[Luigi Toscano (ltoscano)]

Is it the last sahara master? Is it pike? What is the content of sahara.conf? Can you please share some details from your deployment?

[Shu Yingya (felixshu)]

Luigi, this problem affects us from mitaka to stable/pike at least. I don't have master branch to test it right now.
This env is latest RDO deployed by packstack. and the sahara.conf is the default value without any change.

[Rabi Mishra (rabi)]

AFAICT, from heat point of view it should work from newton onwards after https://github.com/openstack/heat/commit/cd125f328e3e4e9f517278b04347515eba380d5a#diff-661fd41a079826552bb7477bd817ea8f landed.

Are you sure you get the above error in pike?

[Shu Yingya (felixshu)]

Hi Rabi, I have retried it on my env (pike RDO) and it can be reproduced 100%.

[Xiaoyang Zhang (es-xiaoyang)]

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+ trust_token +--------+ heat_trust_token +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ 403 error +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :
                                                          heat_trust_token
                         reauthentication_auth_method =trusts +
             user_token +------+ trust_token +--------+ auth_ref +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ complete +----------------+
                                                    \
                                                     \ create_trust
                                            403 error \ +----------+
                                                       -> | keystone |
                                                          +----------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

[Xiaoyang Zhang (es-xiaoyang)]

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+
                        | sahara |
                        +--------+
                            |
                 403 error | heat_trust_token
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+ create_trust +----------+
                        | sahara | ------------> | keystone |
                        +--------+ 403 error +----------+
                            |
                            | heat_trust_token
                   complete | +
                            | auth_ref
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

[Rabi Mishra (rabi)]

> We also need to modify heat to create trust. Add allow_redelegation=True parameter

You can set allow_redelegation=True in [trust] section of keystone.conf[1.]

However, redelegation does not seem to work atm (possibly keystone issue) as discussed in[2].

[1] https://github.com/openstack/keystone/blob/fab399e26cdbe7cffba895f99d7247896ec6cb82/keystone/conf/trust.py#L25

[2] https://bugs.launchpad.net/heat/+bug/1701498

[Wang Bo (chestack)]

Hi Rabi, AFAIK reauthentication_auth_method =trusts was introduced in[1] to defeat token expiry by reauthentication. So if set reauthentication_auth_method = "", the long-running stack creation will be failed by token expiry? Could you help explain it?

[1] https://review.openstack.org/#/c/226384/

[Rabi Mishra (rabi)]

Yeah, that's in heat (we use trusts auth plugin with keystone session which would request for another token if the token expires(401][1]. But if sahara uses heat provided trust token to again request for a token, then it would fail.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726

[Wang Bo (chestack)]

Rabi, thanks for your comment.
From the link you pasted[1], The re-authentication happened in keystone that looks like working for all kinds keystone session(not only trusts auth plugin but also [2]). If that we could leave reauthentication_auth_method = ""(re-authentication will still work). Please correct me if anything wrong.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726
[2] https://github.com/openstack/heat/blob/master/heat/common/context.py#L283-L284

[Rabi Mishra (rabi)]

No it would not work as by default we use the access plugin[1] with the token_info we get from the auth middleware.

The reauth would work only cases where password and trust plugins are used.

[1] https://github.com/openstack/heat/blob/master/heat/common/context.py#L251

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/505476

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/508099

[Lance Bragstad (lbragstad)]

This might be something we can improve in the keystone documentation. It appears keystone has some limitations on what you can do with impersonation and redelegation [0].

Unfortunately, I can't seem to find clear documentation on it. I'll dig a bit more, but this might classify as a documentation fix for keystone.

[0] https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_trust.py#L383-L386

[Vladislav Kuzmin (vkuzmin-u)]

When I run Tempest test I met the same problem when I set reauthentication_auth_method = trusts.

I see in nova-compute.log the following:

2019-04-29 08:21:26,167.167 20630 ERROR castellan.key_manager.barbican_key_manager [req-d8a4f4af-27de-4f2e-b6e0-ff8ec2a11b3c 78f42539a131464ba12e64b683d839d0 14226b3ef05a4b9fa18ca080fe084c2b - default default] Error creating Barbican client: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403) (Request-ID: req-dbc45fcd-5918-4bb2-9f87-40127a2c1607): Forbidden: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403) (Request-ID: req-dbc45fcd-5918-4bb2-9f87-40127a2c1607)

[OpenStack Infra (hudson-openstack)]

Change abandoned by "ruifaling <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/sahara/+/508099
Reason: Too long no progress

[OpenStack Infra (hudson-openstack)]

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/sahara/+/671885
Reason: sahara project has been retired https://review.opendev.org/c/openstack/governance/+/919374