Comment 0 for bug 1496995

The webhook-based trigger workflow is not yet supported in Solum in true end-to-end fashion. We are able to create DUs upon receiving a trigger, but heat deployment fails with 403 (Forbidden) from Heat.
Here is the stack trace from Solum's deployer.

2015-09-17 19:33:34.493 10891 ERROR solum.deployer.handlers.heat [-] ERROR: Remote error: Forbidden Remaining redelegation depth of 0 out of allowed range of [0..3] (Disable debug mode to suppress these
 details.) (HTTP 403) (Request-ID: req-5ee8efb1-6f61-48ba-802f-8ab5c7505a3a)
[u'
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat Traceback (most recent call last):
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat File "/opt/stack/solum/solum/deployer/handlers/heat.py", line 349, in deploy
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat files=get_file_dict)
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat data=kwargs, headers=headers)
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 265, in json_request
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat resp = self._http_request(url, method, **kwargs)
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 220, in _http_request
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat raise exc.from_response(resp)
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat HTTPInternalServerError: ERROR: Remote error: Forbidden Remaining redelegation depth of 0 out of allowed range of [0..3] (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-5ee8efb1-6f61-48ba-802f-8ab5c7505a3a)
2015-09-17 19:33:34.493 10891 TRACE solum.deployer.handlers.heat [u'

The corresponding stack trace from the heat-api:

15-09-17 19:33:34.481 TRACE heat.common.wsgi
2015-09-17 19:33:34.488 DEBUG heat.common.serializers [req-da14c3fd-64d9-4695-9006-6d53480141ec None demo] JSON response : {"explanation": "The server has either erred or is incapable of performing the requested operation.", "code": 500, "error": {"message": "Remote error: Forbidden Remaining redelegation depth of 0 out of allowed range of [0..3] (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-5ee8efb1-6f61-48ba-802f-8ab5c7505a3a)\n[u'", "traceback": "Traceback (most recent call last):\\n', u' File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\", line 142, in _dispatch_and_reply\\n executor_callback))\\n', u' File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\", line 186, in _dispatch\\n executor_callback)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py\", line 129, in _do_dispatch\\n result = func(ctxt, **new_args)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py\", line 105, in wrapper\\n return f(*args, **kwargs)\\n', u' File \"/opt/stack/heat/heat/common/context.py\", line 283, in wrapped\\n return func(self, ctx, *args, **kwargs)\\n', u' File \"/opt/stack/heat/heat/engine/service.py\", line 703, in create_stack\\n stack.store()\\n', u' File \"/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py\", line 105, in wrapper\\n return f(*args, **kwargs)\\n', u' File \"/opt/stack/heat/heat/engine/stack.py\", line 484, in store\\n trust_ctx = keystone.create_trust_context()\\n', u' File \"/opt/stack/heat/heat/common/heat_keystoneclient.py\", line 240, in create_trust_context\\n role_names=roles)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/contrib/trusts.py\", line 76, in create\\n **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py\", line 73, in func\\n return f(*args, **new_kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py\", line 333, in create\\n self.key)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py\", line 151, in _create\\n return self._post(url, body, response_key, return_raw, **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/base.py\", line 165, in _post\\n resp, body = self.client.post(url, body=body, **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py\", line 176, in post\\n return self.request(url, \\'POST\\', **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py\", line 206, in request\\n resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/adapter.py\", line 95, in request\\n return self.session.request(url, method, **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py\", line 336, in inner\\n return func(*args, **kwargs)\\n', u' File \"/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py\", line 397, in request\\n raise exceptions.from_response(resp, method, url)\\n', u'Forbidden: Remaining redelegation depth of 0 out of allowed range of [0..3] (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-5ee8efb1-6f61-48ba-802f-8ab5c7505a3a)\\n'].", "type": "RemoteError"}, "title": "Internal Server Error"} from (pid=20128) to_json /opt/stack/heat/heat/common/serializers.py:42
2015-09-17 19:33:34.491 DEBUG eventlet.wsgi.server [req-da14c3fd-64d9-4695-9006-6d53480141ec None demo] 10.0.2.15 - - [17/Sep/2015 19:33:34] "POST /v1/fd7f57c1ad5743939111144982095cc0/stacks HTTP/1.1" 500 3588 1.004695 from (pid=20128) write /opt/stack/heat/heat/common/wsgi.py:267