heat stack-create failed due to lack of 'v2.0' in auth_uri

[Eric Lee]

When I execute `heat list`, if I have auth_uri = http://keystone_server:5000/v2.0 in heat.conf, it will not pass the authentication in keystone, the message heat-engine.log shows as follow:

2014-09-17 08:50:38.809 334 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2014-09-17 08:50:38.810 334 INFO keystonemiddleware.auth_token [-] Invalid user token - rejecting request

after I revise it to auth_uri = http://keystone_server:5000 and added auth_version = 2.0 then `heat list` acts fine.
However, when I execute `heat stack-create`, it raised the following exception:

2014-09-17 08:42:56.365 32545 TRACE root RemoteError: Remote error: AuthorizationFailure Authorization failed: The resource could not be found. (HTTP 404)
2014-09-17 08:42:56.365 32545 TRACE root [u'Traceback (most recent call last):\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 134, in _dispatch_and_reply\n incoming.message))\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', u' File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py", line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', u' File "/usr/local/lib/python2.7/dist-packages/heat/engine/service.py", line 67, in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File "/usr/local/lib/python2.7/dist-packages/heat/engine/service.py", line 614, in create_stack\n stack.store()\n', u' File "/usr/local/lib/python2.7/dist-packages/heat/engine/stack.py", line 315, in store\n trust_ctx = keystone.create_trust_context()\n', u' File "/usr/local/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 278, in create_trust_context\n trustee_user_id = self.admin_client.auth_ref.user_id\n', u' File "/usr/local/lib/python2.7/dist-packages/heat/common/heat_keystoneclient.py", line 132, in admin_client\n if c.authenticate():\n', u' File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 318, in inner\n return func(*args, **kwargs)\n', u' File "/usr/local/lib/python2.7/dist-packages/keystoneclient/httpclient.py", line 407, in authenticate\n resp = self.get_raw_token_from_identity_service(**kwargs)\n', u' File "/usr/local/lib/python2.7/dist-packages/keystoneclient/v3/client.py", line 266, in get_raw_token_from_identity_service\n \'%s\' % e)\n', u'AuthorizationFailure: Authorization failed: The resource could not be found. (HTTP 404)\n'].

According to the source code, I figure out it was triggered by lack of 'v2.0' in the auth_uri setting, so I revised code in heat/common/heat_keystoneclient.py in KeystoneClientV3.__init__ method, line 78 and line 82:

line 78 before:
self.v3_endpoint = self.context.auth_url.replace('v2.0', 'v3')
line 78 after:
self.v3_endpoint = '/'.join((self.context.auth_url, 'v3'))

line 82 before:
self.v3_endpoint = cfg.CONF.keystone_authtoken.auth_uri.replace(
                'v2.0', 'v3')
line 82 after:
self.v3_endpoint = '/'.join((cfg.CONF.keystone_authtoken.auth_uri, 'v3'))

After doing this, `heat stack-create acts fine.

[Thomas Herve (therve)]

I don't understand. You need to have v2.0 in your auth_uri parameter. So the title of your bug is incorrect, or at least it's the expected behavior.

But in your description, you say that it doesn't work when v2.0 is in the URI. Which is which?

[Eric Lee (eric-lee-ltk)]

Well, in other projects like cinder and swift, v2.0 is not in auth_uri,
however, in heat configuration manual, it recommends the auth_uri contains
v2.0, after adding v2.0 in auth_uri, I encountered the invalid token which
I described in the bug report. If I just left auth_uri without v2.0, at
least I can execute `heat list` with the invalid token warning and acts
fine. However, without v2.0 in auth_uri just can not execute `heat
stack-create` command normally which i described the details in the bug
report. Hence, what i want to express is we should remove the dependence of
v2.0 in the code to get a keystone v3 endpoint. Sorry for my inexact
expression.

2014-10-01 21:02 GMT+08:00 Thomas Herve <email address hidden>:

> I don't understand. You need to have v2.0 in your auth_uri parameter. So
> the title of your bug is incorrect, or at least it's the expected
> behavior.
>
> But in your description, you say that it doesn't work when v2.0 is in
> the URI. Which is which?
>
> ** Changed in: heat
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1370302
>
> Title:
> heat stack-create failed due to lack of 'v2.0' in auth_uri
>
> Status in Orchestration API (Heat):
> Incomplete
>
> Bug description:
> When I execute `heat list`, if I have auth_uri =
> http://keystone_server:5000/v2.0 in heat.conf, it will not pass the
> authentication in keystone, the message heat-engine.log shows as
> follow:
>
> 2014-09-17 08:50:38.809 334 WARNING keystonemiddleware.auth_token [-]
> Authorization failed for token
> 2014-09-17 08:50:38.810 334 INFO keystonemiddleware.auth_token [-]
> Invalid user token - rejecting request
>
>
> after I revise it to auth_uri = http://keystone_server:5000 and added
> auth_version = 2.0 then `heat list` acts fine.
> However, when I execute `heat stack-create`, it raised the following
> exception:
>
> 2014-09-17 08:42:56.365 32545 TRACE root RemoteError: Remote error:
> AuthorizationFailure Authorization failed: The resource could not be found.
> (HTTP 404)
> 2014-09-17 08:42:56.365 32545 TRACE root [u'Traceback (most recent call
> last):\n', u' File
> "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
> line 134, in _dispatch_and_reply\n incoming.message))\n', u' File
> "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
> line 177, in _dispatch\n return self._do_dispatch(endpoint, method,
> ctxt, args)\n', u' File
> "/usr/local/lib/python2.7/dist-packages/oslo/messaging/rpc/dispatcher.py",
> line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
> **new_args)\n', u' File
> "/usr/local/lib/python2.7/dist-packages/heat/engine/service.py", line 67,
> in wrapped\n return func(self, ctx, *args, **kwargs)\n', u' File
> "/usr/local/lib/python2.7/dist-packages/heat/engine/service.py", line 614,
> in create_stack\n stack.store()\n', u' File
> "/usr/local/lib/python2.7/dist-packages/heat/engine/stack.py", line 315, in
> store\n trust_ctx = keystone.create_trust_context()\n', u' File
> "/usr/local/lib/python2.7/dist-pack...

[Thomas Herve (therve)]

I don't know where you got the idea where auth_uri shouldn't contain v2.0. That has always been the case AFAIK. See https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token.py#L240 for example.

I guess the question is why heat list (really should be heat stack-list) doesn't work for you. Can you paste the output of "heat --debug stack-list" when auth_uri contains v2.0? Thanks.

[Steven Hardy (shardy)]

I think this may be a conflation of two issues:

1. Right now, we do need v2.0 as a path suffix for auth_uri - it's fair to say this is suboptimal and should be replaced with version discovery via keystoneclient during Kilo, but Heat (unlike many other projects) makes use of some v3 only keystone functionality (domains specifically), so we have to specify the v3 endpoint directly, rather than using the (v2 versioned in most deployments) catalog endpoint.

2. If after setting v2.0 things aren't working due to invalid user token, is it possible that you're passing a v3 token without setting auth_version=v3.0? In my previous testing I found it was necessary to set v3.0 for keystone auth_token to authenticate correctly with v3 tokens from a non-default domain.

So, while I agree in due course we should remove the v2.0 dependence in the code, if you set it and things still don't work, please provide further information (the backtrace from heat --debug stack-list for example as requested by therve) so we can help debug it.

[Eric Lee (eric-lee-ltk)]

debug messages are as follow when I execute `heat --debug stack-list` with auth_uri=http://keystone_server:5000/v2.0 in heat.conf:

DEBUG (v2) Making authentication request to http://controller:5000/v2.0/tokens
DEBUG (http) curl -i -X GET -H 'X-Auth-Token: b97500ac1d9849bd91d63c5a04b59ea1' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'User-Agent: python-heatclient' http://controller:8004/v1/c8f88a72e4e141f29339a9d30fe93d04/stacks?
DEBUG (http)
HTTP/1.1 401 Unauthorized
date: Wed, 08 Oct 2014 00:35:13 GMT
content-length: 23
content-type: text/plain
www-authenticate: Keystone uri='http://controller:5000/v2.0'
x-openstack-request-id: req-7e0bea9b-77ae-4286-ad6a-675b51e69b72

Authentication required

Traceback (most recent call last):
  File "/usr/local/bin/heat", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/heatclient/shell.py", line 443, in main
    HeatShell().main(args)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/shell.py", line 399, in main
    args.func(client, args)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 432, in do_stack_list
    utils.print_list(stacks, fields, sortby_index=3)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/openstack/common/cliutils.py", line 154, in print_list
    for o in objs:
  File "/usr/local/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 85, in paginate
    stacks = self._list(url, 'stacks')
  File "/usr/local/lib/python2.7/dist-packages/heatclient/openstack/common/apiclient/base.py", line 115, in _list
    body = self.client.get(url).json()
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 259, in get
    return self.client_request("GET", url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 252, in client_request
    resp, body = self.json_request(method, url, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 233, in json_request
    resp = self._http_request(url, method, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/heatclient/common/http.py", line 192, in _http_request
    % resp.content)
heatclient.exc.HTTPUnauthorized: ERROR: Authentication failed. Please try again with option --include-password or export HEAT_INCLUDE_PASSWORD=1
Authentication required

[Launchpad Janitor (janitor)]

[Expired for heat because there has been no activity for 60 days.]

[Desh Shukla (deshdeepak-shukla)]

I tried the following and it worked for me on Juno release

auth_uri=http://keystone_server:5000/v2.0/