Failed to create sahara cluster if reauthentication_auth_method =trusts

Bug #1717918 reported by Wang Bo
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Heat
New
Undecided
Unassigned
OpenStack Identity (keystone)
Triaged
Low
Unassigned
Sahara
New
Undecided
Unassigned

Bug Description

set reauthentication_auth_method =trusts

heat stack-create sahara_cluster -f sahara_cluster.yaml

sahara-engine came into error when to get image, with following error:
Forbidden: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403)

The error log is very clear, heat passed trust scoped token to sahara.
Setting "reauthentication_auth_method =trusts" works well for other components but sahara.

sahara_cluster.yaml:
heat_template_version: 2015-04-30

resources:
  sahara_cluster:
    type: OS::Sahara::Cluster
    properties:
      name: rdo
      plugin_name: spark
      hadoop_version: 1.6.0
      default_image_id: 4ae3ae1a-ca7f-428b-ae2b-782fb81a9dd2
      cluster_template_id: 7c8f8453-0934-4270-917f-7e3188320a9a
      key_name: stack
      neutron_management_network: private
      use_autoconfig: true

Wang Bo (chestack)
affects: heat → sahara
affects: sahara → heat
Revision history for this message
Luigi Toscano (ltoscano) wrote :

Is it the last sahara master? Is it pike? What is the content of sahara.conf? Can you please share some details from your deployment?

Revision history for this message
Shu Yingya (felixshu) wrote :

Luigi, this problem affects us from mitaka to stable/pike at least. I don't have master branch to test it right now.
This env is latest RDO deployed by packstack. and the sahara.conf is the default value without any change.

Revision history for this message
Rabi Mishra (rabi) wrote :

AFAICT, from heat point of view it should work from newton onwards after https://github.com/openstack/heat/commit/cd125f328e3e4e9f517278b04347515eba380d5a#diff-661fd41a079826552bb7477bd817ea8f landed.

Are you sure you get the above error in pike?

Revision history for this message
Shu Yingya (felixshu) wrote :

Hi Rabi, I have retried it on my env (pike RDO) and it can be reproduced 100%.

Revision history for this message
Xiaoyang Zhang (es-xiaoyang) wrote :

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+ trust_token +--------+ heat_trust_token +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ 403 error +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :
                                                          heat_trust_token
                         reauthentication_auth_method =trusts +
             user_token +------+ trust_token +--------+ auth_ref +----------------+
create_stact ----------> | heat | -----------> | sahara | ----------------> | nova/glance... |
                         +------+ +--------+ complete +----------------+
                                                    \
                                                     \ create_trust
                                            403 error \ +----------+
                                                       -> | keystone |
                                                          +----------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

Revision history for this message
Xiaoyang Zhang (es-xiaoyang) wrote :

When reauthentication_auth_method = trusts, heat uses trust token to build sahara client. However, sahara uses heat`s trust token to build nova session client without auth_ref.

nova = nova_client.Client('2', session=session, auth=keystone.auth(),
                          endpoint_type=CONF.nova.endpoint_type,
                          region_name=CONF.os_region_name)

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+
                        | sahara |
                        +--------+
                            |
                 403 error | heat_trust_token
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

This is not allowed in keystone token method. The original is described in this way : 'Do not allow tokens used for delegation to create another token, or perform any changes of state in Keystone. To do so is to invite elevation of privilege attacks'
There are two possible solutions without changing the heat configuration :
1. Sahara uses HTTPclient when building other components of client.
2. When building other components client, add auth_ref.

The problem will have a new problem after it is repaired. Sahara uses heat_trust_token to create new trust. The heat_trust_token`s redelegation_count==0 causes 403 errors.
problem analysis :

                         reauthentication_auth_method =trusts
             user_token +------+
create_stact ----------> | heat |
                         +------+
                            |
                            | trust_token
                            V
                        +--------+ create_trust +----------+
                        | sahara | ------------> | keystone |
                        +--------+ 403 error +----------+
                            |
                            | heat_trust_token
                   complete | +
                            | auth_ref
                            V
                    +----------------+
                    | nova/glance... |
                    +----------------+

We also need to modify heat to create trust. Add allow_redelegation=True parameter :

trust = self.client.trusts.create(trustor_user=trustor_user_id,
                                  trustee_user=trustee_user_id,
                                  project=trustor_proj_id,
                                  impersonation=True,
                                  allow_redelegation=True,
                                  role_names=roles)

Revision history for this message
Rabi Mishra (rabi) wrote :

> We also need to modify heat to create trust. Add allow_redelegation=True parameter

You can set allow_redelegation=True in [trust] section of keystone.conf[1.]

However, redelegation does not seem to work atm (possibly keystone issue) as discussed in[2].

[1] https://github.com/openstack/keystone/blob/fab399e26cdbe7cffba895f99d7247896ec6cb82/keystone/conf/trust.py#L25

[2] https://bugs.launchpad.net/heat/+bug/1701498

Revision history for this message
Wang Bo (chestack) wrote :

Hi Rabi, AFAIK reauthentication_auth_method =trusts was introduced in[1] to defeat token expiry by reauthentication. So if set reauthentication_auth_method = "", the long-running stack creation will be failed by token expiry? Could you help explain it?

[1] https://review.openstack.org/#/c/226384/

Revision history for this message
Rabi Mishra (rabi) wrote :

Yeah, that's in heat (we use trusts auth plugin with keystone session which would request for another token if the token expires(401][1]. But if sahara uses heat provided trust token to again request for a token, then it would fail.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726

Revision history for this message
Wang Bo (chestack) wrote :

Rabi, thanks for your comment.
From the link you pasted[1], The re-authentication happened in keystone that looks like working for all kinds keystone session(not only trusts auth plugin but also [2]). If that we could leave reauthentication_auth_method = ""(re-authentication will still work). Please correct me if anything wrong.

[1] https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/session.py#L724-L726
[2] https://github.com/openstack/heat/blob/master/heat/common/context.py#L283-L284

Revision history for this message
Rabi Mishra (rabi) wrote :

No it would not work as by default we use the access plugin[1] with the token_info we get from the auth middleware.

The reauth would work only cases where password and trust plugins are used.

[1] https://github.com/openstack/heat/blob/master/heat/common/context.py#L251

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to heat (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/505476

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to sahara (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/508099

Revision history for this message
Lance Bragstad (lbragstad) wrote :

This might be something we can improve in the keystone documentation. It appears keystone has some limitations on what you can do with impersonation and redelegation [0].

Unfortunately, I can't seem to find clear documentation on it. I'll dig a bit more, but this might classify as a documentation fix for keystone.

[0] https://github.com/openstack/keystone/blob/master/keystone/tests/unit/test_v3_trust.py#L383-L386

tags: added: office-hours
Changed in keystone:
status: New → Triaged
importance: Undecided → Low
tags: added: documentation
Rico Lin (rico-lin)
Changed in heat:
milestone: none → no-priority-tag-bugs
Revision history for this message
Vladislav Kuzmin (vkuzmin-u) wrote :

When I run Tempest test I met the same problem when I set reauthentication_auth_method = trusts.

I see in nova-compute.log the following:

2019-04-29 08:21:26,167.167 20630 ERROR castellan.key_manager.barbican_key_manager [req-d8a4f4af-27de-4f2e-b6e0-ff8ec2a11b3c 78f42539a131464ba12e64b683d839d0 14226b3ef05a4b9fa18ca080fe084c2b - default default] Error creating Barbican client: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403) (Request-ID: req-dbc45fcd-5918-4bb2-9f87-40127a2c1607): Forbidden: You are not authorized to perform the requested action: Using trust-scoped token to create another token. Create a new trust-scoped token instead. (HTTP 403) (Request-ID: req-dbc45fcd-5918-4bb2-9f87-40127a2c1607)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on sahara (master)

Change abandoned by "ruifaling <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/sahara/+/508099
Reason: Too long no progress

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by "Ghanshyam <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/sahara/+/671885
Reason: sahara project has been retired https://review.opendev.org/c/openstack/governance/+/919374

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.