Workflow execution lifespan is limited by auth token expire time

Bug #1595084 reported by Renat Akhmerov
34
This bug affects 5 people
Affects Status Importance Assigned to Milestone
Mistral
In Progress
High
Unassigned

Bug Description

Now if we run a workflow that will be running longer than a lifetime of an authentication token associated with it then it will fail at some point when the token gets expired.

Keystone trusts are currently used only for cron triggers to be able to run workflows in a deferred mode. But trusts do not solve the aforementioned problem.

This current behavior was implemented intentionally because we didn't want to stick to Identity API of version 3 (that provides trusts) too much at that time since some people may not want to use v3. The assumption was that workflows that users start won't take longer than token expiration period. But as we become more stable and expand our use cases to more complicated things that require longer run time it's becoming an issue for us.

Some thoughts on possible ways to solve it:
* Use trusts not only for cron triggers and get a new token during workflow execution from trust before running every action in the workflow because actions may require a new token (e.g. those that interact with OpenStack services).
* Same as before but we can implement smarter logic that will determine dynamically if token needs to be renewed (e.g. if it's about to get expired).
* Somehow dynamically determine if next action needs a token and renew it, if needed.

In general, this problem comes down to having to have some token renewal mechanism in Mistral.

If authentication is disabled, this bug is not relevant.

Tags: tripleo
Changed in mistral:
importance: Undecided → High
status: New → Confirmed
Revision history for this message
Andras Kovi (akovi) wrote :

Regarding the solutions:

2) and 3)
- the actions invoked can also time out internally, we have no influence on this.
- E.g. call a heat stack create that takes who knows much time. How do you decide whether the token needs to be renewed before the action? What if the token expiration is shorter than the expected length of the operation?
- renewing a token without admin credentials is impossible

Due to these reasons, I think 2) and 3) are not valid solutions.

1) yes, I think we should go for using trusts
- maybe the Mistral client could create the trust and Mistral could simply reuse it
- creating trusts does not require admin rights
- Mistral could run without admin credentials which is very important for standalone applications

Finally, IMHO at this point requiring keystone V3 should not be a limitation for deployments.

Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

Andras,

Regarding your comments on 2) and 3), You may be right but I'm not sure on 100%. Renewal is assumed anyway, that's why trust is needed. And it seems to me that if we were able to make a call to OpenStack with some token then we're good, regardless token expiration time, no? Otherwise, for example, if token expiration period is 3 days and I start creating a heat stack which takes 4 days, it will still fail. If so, I can always come up with a big enough stack that will fail to create regardless of my token expiration period. What am I missing?

Btw, I thought that access impersonation (in our case using trust) does not require admin permissions. Based on a user token, Heat just creates another trust which in turn is used to get other tokens inside Heat with the same permissions as the original token. Why would we need admin access here? For what?

Renat

Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

On v3 agree, it is not a problem any more.

Revision history for this message
Andras Kovi (akovi) wrote :

Renat,

I don't know exactly how Heat uses trusts. But I'd assume that it will pass that trust on to other services that can use it to renew the token and prevent the long operation from failing. So it should not matter how long an operation takes. As long as the trust is valid, all services south from me (callstack wise) should be able to perform the operation I told them to do. So a 4 day operation with 3 day validity and a trust should still succeed in my opinion.

I mentioned the admin permission because Mistral currently uses it. It creates the trust for the admin user in the cron jobs. If this was removed, there would be no need for the admin credentials in the Mistral setup. (The service endpoint listing is being de-adminized by https://review.openstack.org/#/c/330469)

Andras

Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

Ok, honestly I'm not well aware of recent changes in trusts (e.g. there's now trust http header that can be passed through) so I'd like to learn more on this first. We'll need to look at all that stuff when working on this ticket.

Using admin creds in Mistral seems to be unnecessary to me. I would like to get rid of it. It needs to be investigated.

Thanks

Revision history for this message
Steven Hardy (shardy) wrote :

Hi all, I implemented the workaround for this in heat, which does use trusts. I'm very happy to discuss how we did that, and potentially to help fix this bug (we can perhaps reuse some of the code).

The basic approach is:

1. User creates a heat stack, heat creates a trust delegating the users roles to the heat service user
2. Inside heat we have a wrapper around the current token, and each time it's accessed we check the token expiry time
3. If the token is about to expire, we create another token, with the trust, so we're impersonating the user and have a new token where we can act on their behalf.

I discussed it with the keystone team, and was advised that this is a valid use of Trusts, and it works well.

Here is the code:

Creating the trust:

https://github.com/openstack/heat/blob/master/heat/engine/clients/os/keystone/heat_keystoneclient.py#L210

Checking for token expiry:

https://github.com/openstack/heat/blob/master/heat/common/context.py#L164

Forcing a new token:

https://github.com/openstack/heat/blob/master/heat/common/context.py#L151

As you can see, we wrap some of the keystone session/plugin logic in the context, which is convenient as it enables it to be easily shared with the various openstack clients (which are also wrapped in a plugin interface) heat needs. I think the mistral requirements are similar here, but I've not yet looked at the code to see how similar your client usage is.

If you're open to some patches, I'm happy to take a closer look and/or discuss further (shardy on IRC), thanks!

Revision history for this message
Renat Akhmerov (rakhmerov) wrote :

Steve, thanks a lot for taking participation. The algorithm you described makes perfect sense to me. Once we start with the implementation (hopefully in Aug) we'll be contacting you, if needed. Currently, it's not been started yet.

Very useful information. Thanks again.

Revision history for this message
Andras Kovi (akovi) wrote :

Steve, just one question:

Could you clarify if this requires an admin client on the Heat side? Does the trust creation work without an admin account?

We are aiming to make Mistral more general and standalone. In such a setup Mistral is expected to execute actions on behalf of the user but will not have admin rights to the cloud.

Thanks in advance!

Revision history for this message
Steven Hardy (shardy) wrote :

> Could you clarify if this requires an admin client on the Heat side? Does the trust creation work without an admin account?

No, you just require two users, one making the API request to heat/mistral (the trustor), and some user associated with heat/mistral to delegate to (the trustee) - neither user has to be admin.

That said, by default in the heat case we do delegate to the heat service user, which typically has the admin role, but this is just for convenience as the user already exists, you could create a separate trustee user without admin rights (and this would probably be a good idea from a security standpoint).

Revision history for this message
luong tuan (tuantuluong) wrote :

AFAIK, now in keystone we have some supports for expired tokens as "allow expired token" or "service token" but these features are only supported from current versions of keystone (from octaka). But in some cases, we need to support the older releases that do not support expired token. Therefore IMHO, using trust as heat does is a good way to go.

@Shardy: If you have time, i would like to have a discussion with you about it meanwhile i am trying to implement this idea to mistral.

Changed in mistral:
milestone: none → pike-2
assignee: nobody → luong tuan (tuantuluong)
Changed in mistral:
status: Confirmed → In Progress
Revision history for this message
Moshe Elisha (melisha) wrote : RE: Mistral intro presentation

[Adding other email addresses of Renat]

We have these Mistral training slides https://nokia-my.sharepoint.com/personal/terry_niblack_nokia_com/Documents/Shared/CBND%2017.5%20BootCamp/Day_4/CBND%2017.5%2013%20-%20Mistral%20(for%20custom%20workflows).pptx?d=wfb5f64db4a824d7a9780b9c5bf0217fd

From: Kovi, Andras 1. (Nokia - HU/Budapest)
Sent: Thursday, June 01, 2017 11:46 AM
To: Renat Akhmerov <email address hidden>; Elisha, Moshe (Nokia - IL/Kfar Sava) <email address hidden>
Subject: Fw: Mistral intro presentation

Again, with a better email address. 😊

________________________________
Feladó: Kovi, Andras 1. (Nokia - HU/Budapest)
Elküldve: 2017. június 1. 10:44
Címzett: Akhmerov, Renat (EXT - RU); Elisha, Moshe (Nokia - IL/Kfar Sava)
Tárgy: Mistral intro presentation

Hi Renat, hi Moshe,

I hope you're doing well. Next week I present the Mistral performance at the Openstack Day here in the HQ. I think it would be necessary to add a few introductory slides about the service. Could you send me a few 'canned' ones. It would be tremendously helpful.

Thank you,

Andras

Changed in mistral:
milestone: pike-2 → pike-3
Changed in mistral:
milestone: pike-3 → pike-rc1
Changed in mistral:
milestone: pike-rc1 → queens-1
Changed in mistral:
milestone: queens-1 → queens-3
Changed in mistral:
milestone: queens-3 → rocky-1
Revision history for this message
Dougal Matthews (d0ugal) wrote :

Removed assignee as there has been no recent activity.

Changed in mistral:
assignee: luong tuan (tuantuluong) → nobody
status: In Progress → Triaged
Dougal Matthews (d0ugal)
tags: added: tripleo
Brad P. Crochet (brad-9)
Changed in mistral:
assignee: nobody → Brad P. Crochet (brad-9)
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-1 → rocky-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to mistral (master)

Fix proposed to branch: master
Review: https://review.openstack.org/572448

Changed in mistral:
status: Triaged → In Progress
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-2 → rocky-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on mistral (master)

Change abandoned by Brad P. Crochet (<email address hidden>) on branch: master
Review: https://review.openstack.org/572448
Reason: Will see if Change-Id: Ia2a19a3fcd8808475a16d4d439e085e62a00dfdc works better

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Dougal Matthews (<email address hidden>) on branch: master
Review: https://review.openstack.org/462056
Reason: Abandoning in favor of...
https://review.openstack.org/#/c/572448/
https://review.openstack.org/#/c/585904/

Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-3 → stein-1
milestone: stein-1 → rocky-rc1
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-rc1 → rocky-rc2
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: rocky-rc2 → stein-1
Dougal Matthews (d0ugal)
Changed in mistral:
milestone: stein-1 → stein-2
Brad P. Crochet (brad-9)
Changed in mistral:
assignee: Brad P. Crochet (brad-9) → nobody
Changed in mistral:
milestone: stein-2 → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Change abandoned by Brad P. Crochet (<email address hidden>) on branch: master
Review: https://review.opendev.org/572448

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.