Possible confusion between auth_uri and auth_port

[Jordan Pittier]

Hello,
When Heat tries to contact Keystone, it reads the configuration flag keystone_authtoken::auth_uri

According to other openstack projects (See [1] [2] and [3]) auth_uri is only meant to be return to client as part of a HTTP-Header. The combination of auth_host and auth_port should be used to validate and ask new token from Keystone. auth_host should point to the admin url of keystone.

At the moment, Heat can't be install in an environment where the publicURL (the one referenced by auth_uri) is not reachable from the Heat engine. Other Openstack projects deals fine with this network topology.

This bug report relates to : https://bugs.launchpad.net/heat/+bug/1259968

[1] http://lists.openstack.org/pipermail/openstack-dev/2013-August/012886.html
[2] https://review.openstack.org/#/c/39796/
[3] https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L191

[Steven Hardy (shardy)]

So there appear to be two issues here:

1. For both the core and contrib v2 keystoneclient wrapper, we use auth_uri for the auth_url passed into keystoneclient.

Arguably, as reported, this should instead use the auth_host/auth_port to connect to the admin endpoint for connections made as an admin (e.g when not using auth_url from the context, i.e using X-Auth-Url in standalone mode)

2. In the core keystoneclient wrapper, we override what is in the keystone catalog by passing auth_url and endpoint

This was to enable reliable access to v3 keystone functionality in environments where the catalog endpoints were versioned as v2.0. It may be that we can remove this now, as I believe work has been completed in keystoneclient to enable version discovery - hopefully this means we can move to just passing the auth_url and letting the client work out the endpoint from the catalog.

In both cases, my main concern is breaking existing functionality, we'll have to do some testing and figure out what works.

An alternative, simpler/less-risky solution is to provide a heat-specific configuration for admin connections to keystone. Arguably we're doing the wrong thing by reusing API configuration in the engine so perhaps having this (optionally) decoupled in the config file makes sense.

[Jamie Lennox (jamielennox)]

This problem I think can be generalized to heat (or anyone) should not use, rely on or in any way touch the configuration of auth_token middleware. This configuration is used specifically for token validation and heat shouldn't need to be reacting when the config options of auth_token middleware changes.

[Adam Gandelman (gandelman-a)]

@Jamie-

I discovered devstack is currently setting incorrect/obsolete auth_token options [1] The issue cropped up for me in Ironic, which contains a code path that pulls out specific config options from the auth_token section [2] Your comments here make it sound like that is an Ironic bug, and that it should be creating a client in some other way. Thoughts?

[1] https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/keystone#n425 (ie, username instead of admin_user)
[2] https://git.openstack.org/cgit/openstack/ironic/tree/ironic/common/keystone.py#n63

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/164274
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=a8e8963b6868948c875545e5bfdeeb6e3511a1ae
Submitter: Jenkins
Branch: master

commit a8e8963b6868948c875545e5bfdeeb6e3511a1ae
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Mar 13 20:19:03 2015 +0300

    Changing method for verifying existence of cinder

    Sahara uses keystone admin client for checking existence of cinder.
    This patch change method for checking existence of cinder

    Change-Id: I082c2392742476eb7fa1e74d2443d10c89d6f707
    Partial-bug: #1300246

[OpenStack Infra (hudson-openstack)]

Change abandoned by Sergey Lukjanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/163873
Reason: This path is older then 3 month, abandoned

[Matt Kassawara (ionosphere80)]

This issue continues to cause major headaches during installation guide development and probably any deployment of heat, especially after keystone middleware deprecated the identity_uri, admin_tenant_name, admin_user, and admin_password options. Reuse of options aside, heat should at least support the following options (example values shown) in the [keystone_authtoken] section:

auth_uri = http://controller:5000 (note lack of version here)
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = heat
password = heatpass

[Steven Hardy (shardy)]

@Matt Kassawara: Thanks for raising this, however please can you clarify your comment #6 - AIUI heat should not be using the keystone_authtoken config, as mentioned in comment #2, so we now support a separate "clients_keystone" section instead, which is completely independent of the keystone_authtoken section.

To avoid breaking existing users we still support a fallback mode to use keystone_authtoken, but it is no longer required.

However you appear to be requesting action related to the keystone_authtoken config, which, AIUI, should not be required given that section is exclusively for use by keystonemiddleware?

If you can further clarify exactly what major headaches heat is causing I'll be glad to revive this and work to resolve them, but I was under the impression this had been pretty much indirectly fixed via other bugs.

https://github.com/openstack/heat/blob/master/heat/common/config.py#L299

[Matt Kassawara (ionosphere80)]

Oops... probably should provide additional comments here rather than our e-mail thread.

The example configuration file from "tox -e genconfig" only includes the auth_uri option in the [clients_keystone] section. Only using this option in my environment yields a 400 error from heat.common.wsgi with the following message:

BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

I can resolve this issue only by adding the admin_tenant_name, admin_user, and admin_password options in the [keystone_authtoken] section. However, keystone middleware deprecated these options in Kilo and no other services require them.

Additionally, the heat.conf file in the heat gate jobs does not implement [clients_keystone] and continues to use the auth_uri and deprecated admin_* options in the [keystone_authtoken] section.

[Steven Hardy (shardy)]

@Matt: Thanks for the update.

I'll have to test to check, but I think that error is coming from using the auth_token user as the trustee for deferred authentication (keystone trusts).

That is also fixed, by the addition of a new "trustee" config group:

https://github.com/openstack/heat/blob/master/heat/common/context.py#L156

So, I think, the main issue is devstack and any docs need updating to reflect this.

I've targetted this at M1 so we can ensure the devstack default config switches over to avoid using any deprecated interfaces, which I hope will address your concerns.

[Matt Kassawara (ionosphere80)]

Why doesn't the trustee group and options appear in the configuration file?

[Steven Hardy (shardy)]

> Why doesn't the trustee group and options appear in the configuration file?

I think the answer is because the new section was added via a keystoneclient auth plugin, which has it's own config registration handle.

I need to sync up with those involved with fixing https://bugs.launchpad.net/heat/+bug/1446918, then we can work out how to fix up the config generation to include the new section.

[Matt Kassawara (ionosphere80)]

The keystone client or middleware?

[Steven Hardy (shardy)]

https://github.com/openstack/heat/blob/master/heat/common/context.py#L38

The new group is added via keystoneclient.auth - I'm not familiar with why that's done like that, and neither is miguelgrinberg the co-author of the patch.

I'll check with Jamie Lennox, since he invented the auth-plugin mechanism in keystoneclient IIRC, and wrote much of the heat code porting to use them. He'll either be able to clarify how that's supposed to work with genconfig, or confirm that we can convert to directly registering directly via cfg.CONF.

[Steven Hardy (shardy)]

Ok, sorry for the delay on this - I'm now working on fixing the sample config generation so it includes the "trustee" section, and fixing up devstack so it uses that by default.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/254145

[Steven Hardy (shardy)]

I've proposed a patch which adds the keystoneclient options for the "trustee" section to the output from tox -e genconfig.

In summary, you can maintain the previous behavior, which is to delegate to the heat service user defined in the auth_token section by adding this to your heat.conf

[trustee]
auth_plugin = v3password
auth_url = http://<keystone hostname or IP>:5000/v3
username = heat
password = password
user_domain_id = default

Note, this can be maintained independently of any auth_token section, so I believe when that lands, we can close this bug?

I'll also work on some better docs describing how this all works, and update devstack to match this configuration by default.

[Matt Kassawara (ionosphere80)]

Can [trustee] use" auth_plugin = password" with "auth_url" lacking a version similar to how the installation guide uses the password plug-in for all services? For example, glance [1]. Also, is "auth_uri" necessary? Here's what the installation guide currently uses for the [trustee] section [2]:

[trustee]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = heat
password = HEAT_PASS

[1] http://docs.openstack.org/liberty/install-guide-ubuntu/glance-install.html#install-and-configure-components
[2] http://docs.openstack.org/liberty/install-guide-ubuntu/heat-install.html#install-and-configure-components

[Steven Hardy (shardy)]

@Matt:

So, I did some more testing, and it appears that using an unversioned auth_url, and the generic "password" auth_plugin does the right thing:

[trustee]
auth_plugin = password
auth_url = http://192.168.1.14:5000
username = heat
password = foobar
user_domain_id = default

I do not believe the auth_uri entry is necessary at all - I've not used it for testing and I can't see it referenced anywhere in keystoneclient/auth/* (e.g the client plugins which consume this section)

I believe the project_name and project_domain_id in the docs is wrong - in fact I'm pretty sure including those will break completely with the default setting of deferred_auth_method=trusts, because we want the client to get a trust-scoped token, and it can't be scoped to both a trust and the project.

Testing proves that - keystone gives "Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped" if you try the above.

So, we've got some docs errors to work out - thanks for highlighting them.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/254327

[Steven Hardy (shardy)]

Matt: Docs patch proposed which I believe resolves the errors you mentioned - please provide feedback on the review if you're happy it addresses your concerns, thanks!

[Steven Hardy (shardy)]

It's somewhat hard to know the best approach in terms of sample config generation - from the heat perspective all we define is a "trustee" config group, and the required contents of that are entirely dependent on keystoneclient, and which auth_plugin is selected.

So, arguably we should only be documenting the "auth_plugin" and "auth_section" options which are common to all plugins, only that makes the sample config less self documenting than what I've proposed in https://review.openstack.org/#/c/254145/, e.g there we accept that the only known plugin that works in this case is a password plugin that supports trusts, but in theory other plugins could work provided trust scoping is possible (we pass trust_id internally)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254327
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=98ef59a59973954f5396df4898ffc1d1df98b9c2
Submitter: Jenkins
Branch: master

commit 98ef59a59973954f5396df4898ffc1d1df98b9c2
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 17:33:42 2015 +0000

    [install] Fix heat trustee configuration

    The current example config is broken with the default deferred
    authentication of trusts - the project_domain_id and project_name
    cannot be specified in this section or keystone will throw a
    "cannot be scoped to multiple targets" error when we attempt to get
    a token scoped to a trust.

    Also, the auth_uri is unused by the keystoneclient password auth
    plugins, so remove it (it has been a source of confusion in the
    referenced bug).

    Change-Id: I6305be3c693993de4d16d82fd6de936e92c437c5
    Closes-Bug: #1300246
    Backport: liberty

[Steven Hardy (shardy)]

I pushed a patch to devstack which aims to align the default devstack heat.conf with the changes discussed here and in the docs:

https://review.openstack.org/254755

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/254145
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=67f12e5e24eb9ec06e609f4399008b9d704128a1
Submitter: Jenkins
Branch: master

commit 67f12e5e24eb9ec06e609f4399008b9d704128a1
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 11:20:25 2015 +0000

    Add trustee config group to sample config

    Some time ago, we added support for a new "trustee" config section,
    aimed at allowing independent configuration of the credentials used
    to obtain trust-scoped tokens for deferred authentication. One of the
    main reasons for this was to avoid incorrectly using the keystone
    auth_token section, and as such a deprecation warning was added when
    we fall back to using that config section for heat.

    Unfortunately we didn't capture this new section in the sample config
    because it's registered via keystoneclient.auth, so this adds support
    for this section to the sample config generated via tox -e genconfig,
    and adds some notes clarifying usage to the auth_plugin option.

    To move to the new config syntax, but maintain the current behavior,
    which is to delegate to the heat service user, you can add this section
    to your heat.conf

    [trustee]
    auth_plugin = password
    auth_url = http://<keystone hostname or IP>:35357
    username = heat
    password = password
    user_domain_id = default

    The generated config documents many more options, all of those supported
    by the keystoneclient v3 Password auth plugin, but these are the minimum
    to enable delegation to the heat service user in the default domain.

    In new deployments this could be set to some other user (such as one created
    in the heat domain), but note that the trustee should not be changed for
    existing deployments where stacks exist, as the trust stored inside heat
    defines a relationship between the stack owner (trustor) and a specific
    trustee (which will be the heat service user if the deployment is using
    the deprecated path that steals credentials from keystone auth_token).

    Change-Id: I30aeb765a2246ce54b10972ae7187655d85cde1f
    Partial-Bug: #1300246

[Liyingjun (liyingjun)]

When deploying with devstack/keystone v3, the configuration is in keystone_authtoken section is still using admin_tenant_name,admin_password,admin_user, and this causes error: Identity response: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/261398

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/261398
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=44ee7e307b49c5ee9e30ba380e36576189efdc34
Submitter: Jenkins
Branch: master

commit 44ee7e307b49c5ee9e30ba380e36576189efdc34
Author: liyingjun <email address hidden>
Date: Fri Dec 25 10:32:14 2015 +0800

    Fix heat config when using keystone v3

    Use configure_auth_token_middleware to generate keystone_authtoken
    section configration.

    Change-Id: I87e0e60afb958683add2aff2552d26cbf8c9e374
    Closes-bug: #1300246

[Steven Hardy (shardy)]

The heat patch landed, and devstack/docs are updated, so I think the heat part of this can be marked fixed - please let me know if you think there is anything further we need to do.

[Matt Kassawara (ionosphere80)]

The patch works for me.

[Sergey Kraynev (skraynev)]

Change status to Fix Released (now it's a correct status for fixed bug)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270598

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/270598
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=5a3618d4f51a64cc9ef16d9365aa7190b3f5914e
Submitter: Jenkins
Branch: stable/liberty

commit 5a3618d4f51a64cc9ef16d9365aa7190b3f5914e
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 17:33:42 2015 +0000

    [install] Fix heat trustee configuration

    The current example config is broken with the default deferred
    authentication of trusts - the project_domain_id and project_name
    cannot be specified in this section or keystone will throw a
    "cannot be scoped to multiple targets" error when we attempt to get
    a token scoped to a trust.

    Also, the auth_uri is unused by the keystoneclient password auth
    plugins, so remove it (it has been a source of confusion in the
    referenced bug).

    Change-Id: I6305be3c693993de4d16d82fd6de936e92c437c5
    Closes-Bug: #1300246
    Backport: liberty
    (cherry picked from commit 98ef59a59973954f5396df4898ffc1d1df98b9c2)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

[Marianne Linhares Monteiro (mariannelinharesm)]

Is this bug valid for Sahara?

[Telles Mota Vidal Nóbrega (tellesmvn)]

Jeremy Freudberg, does your fix with trustee fixes this issue as well?
https://review.openstack.org/#/c/524936/

[Jeremy Freudberg (jfreud)]

As of https://review.openstack.org/#/c/524936/ , Sahara does not rely on or in any way touch the configuration of auth_token middleware.

So, for Sahara, the bug is resolved.