Heat stack update fails due to Authorization error in deployer

[Arati Mahimane]

On git push, the trigger workflow gets called. The worker creates a new image and calls the deployer for deploying the heat stack. Assembly is getting stuck in Building state since the deployer is not able to update the heat stack.
This error is seen after the trusts code got merged.

Following is the error trace in deployer:

{"explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": {"message": "Property error : compute: image Authorization failed: You are not authorized to perform the requested action. (HTTP 403)", "traceback": "Traceback (most recent call last):\n\n File \"/opt/stack/heat/heat/engine/service.py\", line 62, in wrapped\n return func(self, ctx, *args, **kwargs)\n\n File \"/opt/stack/heat/heat/engine/service.py\", line 598, in update_stack\n updated_stack.validate()\n\n File \"/opt/stack/heat/heat/engine/parser.py\", line 356, in validate\n raise ex\n\nStackValidationFailed: Property error : compute: image Authorization failed: You are not authorized to perform the requested action. (HTTP 403)\n", "type": "StackValidationFailed"}, "title": "Bad Request"}
 log_http_response /opt/stack/python-heatclient/heatclient/common/http.py:122
2014-05-07 22:23:12.537 5823 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: ERROR: Property error : compute: image Authorization failed: You are not authorized to perform the requested action. (HTTP 403)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/solum/solum/deployer/handlers/heat.py", line 112, in deploy
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher parameters=parameters)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/python-heatclient/heatclient/v1/stacks.py", line 126, in update
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher data=kwargs, headers=headers)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/python-heatclient/heatclient/common/http.py", line 235, in json_request
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher resp = self._http_request(url, method, **kwargs)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/python-heatclient/heatclient/common/http.py", line 203, in _http_request
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher resp = self._http_request(path, method, **kwargs)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher File "/opt/stack/python-heatclient/heatclient/common/http.py", line 196, in _http_request
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher raise exc.from_response(resp)
2014-05-07 22:23:12.537 5823 TRACE oslo.messaging.rpc.dispatcher HTTPBadRequest: ERROR: Property error : compute: image Authorization failed: You are not authorized to perform the requested action. (HTTP 403)

[Adrian Otto (aotto)]

@aratim: how long after the trigger is defined are we accessing it? Are we storing trust tokens and reusing them, or generating a new one each time a trigger is accessed? The reason I ask this is to potentially rule out the possibility of an expired token.

[Angus Salkeld (asalkeld)]

Is with the vm or docker build_app? - I am guessing docker.

[Arati Mahimane (arati-mahimane)]

@Adrian: I accessed the trigger within 5 mins after it was created, so the token should not be expired.

[Arati Mahimane (arati-mahimane)]

@Angus: yes, this is with the docker build_app

[Angus Salkeld (asalkeld)]

reproductible with: paste.openstack.org/show/79533/

[Angus Salkeld (asalkeld)]

so the trust token is working well with heat, but when heat uses novaclient it (novaclient) seems to want to re-authenticte.
I am getting the Forbidden() from here: https://github.com/openstack/keystone/blob/master/keystone/auth/plugins/token.py#L47

[Angus Salkeld (asalkeld)]

(chat with Steven Hardy) - looks like we need to do some work in heat to fix this.

<shardy> basically you want to get a token, scoped to a trust, then pass that token around and have it reused as opposed to getting a new token
 You can check if the token is trust scoped:
 https://github.com/openstack/heat/blob/master/heat/common/heat_keystoneclient.py#L173
<asalkeld> shardy, 4 lines up we are calling authenticate
 what if that is already a trust token
<shardy> You mean in _v3_client_init?
<shardy> In heat that never happens, because we only ever consume a trust with username/password
 if you have a context which already contains a trust scoped token, that may be the issue
<asalkeld> shardy, but i am using heat with a trust token
<shardy> Ok that is the problem
<shardy> you need chained delegation, which last time I looked was not implemented in keystone
<asalkeld> ga
<shardy> https://blueprints.launchpad.net/keystone/+spec/trusts-chained-delegation
 hmm, that says implemented
<shardy> https://review.openstack.org/#/c/56243/
 Hmm, that patch doesn't implement what I described in the BP at all :
<asalkeld> so why can't it just happily pass around the trust token?
 (dare I ask)
<shardy> because heat needs to create a trust too, so we can impersonate the user you're already impersonating for deferred operations like autoscaling
 and we can't store the token, because it will expire
<shardy> asalkeld: can you explain the use-case, why do you need to use trusts rather than the user's token in this case?
<asalkeld> so we don't have to store a token that will timeout
<shardy> doh
<asalkeld> this is a git hook triggered action
<asalkeld> and we want to kick off a heat update
<shardy> Hmm, so in that case, can you do the intitial stack create with a normal token, and just the stack update with a trust scoped one?
<asalkeld> shardy, that's what we do
<shardy> asalkeld: Ok, we can probably fix it in heat then
<asalkeld> what do I need to do?
<shardy> if we just use the existing trust-scoped token it should work
 which I think is what you said above, but I misunderstood you
<asalkeld> so I have my trust scoped token and heat needs to figure out to then use it's trust scoped token
<shardy> asalkeld: maybe reversing the if self.context.trust_id and elif self.context.auth_token would be enough
<shardy> I seem to recall there was a reason for that ordering but I can't quite remember what
* shardy will need to do some testing
<asalkeld> shardy, i am happy to do the work if you can give some guidance
<shardy> the times when we set the trust_id in the context should be when we start with an empty context, so I *think* we should be able to let the token take precedence
 we'll need to ensure there's nowhere where we just set the trust_id on the request context though

[Steven Hardy (shardy)]

I've been looking into this, a fix may be possible by modifying heat_keystoneclient to avoid the authenticate() when we find a token in the request. I had initially thought that would mean heat could not create the trust it needs, but having done some testing it seems that may not be the case.

Further info/patch coming soon, hopefully.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/96452

[Steven Hardy (shardy)]

Patch proposed to heat - I've written a little script which I think reproduces your issue, but if you could test that and provide feedback on the review, that would be much appreciated, thanks!

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/97568

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/97569

[Steven Hardy (shardy)]

So, with those two patches, it should now be possible to use a trust-scoped token for all heat actions other than create.

To allow create to work we need to do one or more of:
- Always create a stack with the initial non-trust scoped token (even if it's empty)
- Implement explicit support for trust chaining in keystone (I'm looking into this)
- Add a field to the create body, which enables a pre-created trust_id to be passed in (this would require either Solum to create the trust instead of heat and pass it in, which seems conceptually messy but would probably work).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97568
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=dee6ad112b189f3df1fdaac53d93c904f3536c3c
Submitter: Jenkins
Branch: master

commit dee6ad112b189f3df1fdaac53d93c904f3536c3c
Author: Steven Hardy <email address hidden>
Date: Mon Jun 2 21:52:56 2014 +0100

    Add auth_token_info to request context

    The auth_token (and other) middleware adds a keystone.token_info value
    to the request environment after validating the token. We can pass this
    value via the request context which allow us to avoid always re-requesting
    another token in the keystoneclient instance in heat-engine.

    Change-Id: Icac0e73fbddd5011fb41bd644036ba489189c405
    Partial-Bug: #1317293

[OpenStack Infra (hudson-openstack)]

Change abandoned by Steven Hardy (<email address hidden>) on branch: master
Review: https://review.openstack.org/96452
Reason: Mostly superseded by https://review.openstack.org/#/c/97569/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/97569
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=535dd4e77e2c439a8611f27e5e9fa821cc8d7589
Submitter: Jenkins
Branch: master

commit 535dd4e77e2c439a8611f27e5e9fa821cc8d7589
Author: Steven Hardy <email address hidden>
Date: Tue Jun 3 15:30:15 2014 +0100

    Use auth_token_info to initialize heat_keystoneclient sessions

    Now we have the token info available in the request context, we can
    avoid always re-requesting another token every time a client is
    created.

    This should be better for performance as it avoids one extra request to
    keystone, and it also enables trust-scoped tokens to be more easily
    used, since these cannot be used to request another token, thus
    authentication will fail with the current scheme.

    Partial-Bug: #1317293
    Change-Id: I18955d79c4c760ff3fdd47176a2a54c931277121

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/99338

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99338
Committed: https://git.openstack.org/cgit/stackforge/solum/commit/?id=286bf3a2b3ab5f0934729320bef0fa2c99003115
Submitter: Jenkins
Branch: master

commit 286bf3a2b3ab5f0934729320bef0fa2c99003115
Author: Angus Salkeld <email address hidden>
Date: Thu Jun 12 13:49:30 2014 +1000

    Use auth_token_info to initialize solum_keystoneclient sessions

    Now we have the token info available in the request context, we can
    avoid always re-requesting another token every time a client is
    created.

    This should be better for performance as it avoids one extra request to
    keystone, and it also enables trust-scoped tokens to be more easily
    used, since these cannot be used to request another token, thus
    authentication will fail with the current scheme.

    This is needed for the builder to be used from a mistral task.

    Change-Id: Ic03e315a7a7fb29bd8a499032b61befea260ab7e
    Partial-Bug: #1317293

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/99729

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/99769

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99729
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a75d3c94d8e07a5a73e008925eb6fc25515d7c13
Submitter: Jenkins
Branch: master

commit a75d3c94d8e07a5a73e008925eb6fc25515d7c13
Author: Steven Hardy <email address hidden>
Date: Thu Jun 12 14:18:58 2014 +0100

    Allow passing an existing user_creds_id to parser.Stack

    Currently we always create a new user_creds entry every time a stack
    is stored, even if a user_creds_id is passed to the constructor. So
    allow an existing user_creds ID to be respected.

    This will enable optimisation of the creation of trusts, e.g so we won't
    have to create a new trust for every nested stack, we can pass the top
    level credentials into the nested stack instead, which means updates which
    create nested stacks with a trust-scoped-token will work.

    This will be required for updates via trusts from Solum, and probably
    also to make OS::Heat::AutoScalingGroup work with the trust-alarm-urls
    blueprint, where ceilometer will send a signal using a trust.

    Change-Id: I7f78a2202514f13deb60ad7d66b9145447cc5625
    Partial-Bug: #1317293
    blueprint: trust-alarm-urls

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/99769
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=3cf75a130b77f39c639bca037211b07c2ec995b2
Submitter: Jenkins
Branch: master

commit 3cf75a130b77f39c639bca037211b07c2ec995b2
Author: Steven Hardy <email address hidden>
Date: Tue Jun 3 15:30:15 2014 +0100

    Use auth_token_info to initialize heat_keystoneclient sessions

    Now we have the token info available in the request context, we can
    avoid always re-requesting another token every time a client is
    created.

    This should be better for performance as it avoids one extra request to
    keystone, and it also enables trust-scoped tokens to be more easily
    used, since these cannot be used to request another token, thus
    authentication will fail with the current scheme.

    Change-Id: Id34a4104cedbd3f1c231bc0b9229a5cac7bb9402
    Partial-Bug: #1317293

[Steven Hardy (shardy)]

I believe the heat part of this is now complete - IRC feedback from asalkeld indicated that with the fix for bug #1342593 and the patches proposed for this bug, he was able to successfully update with a trust-scoped token, so marking the heat part of this as fix committed.

Angus, please let me know if you think there are remaining issues and we can capture them in a follow-up bug.