Possible confusion between auth_uri and auth_port

Bug #1300246 reported by Jordan Pittier
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
High
Steven Hardy
Sahara
Fix Released
Medium
Unassigned
devstack
Undecided
Liyingjun
openstack-manuals
Undecided
Matt Kassawara

Bug Description

Hello,
When Heat tries to contact Keystone, it reads the configuration flag keystone_authtoken::auth_uri

According to other openstack projects (See [1] [2] and [3]) auth_uri is only meant to be return to client as part of a HTTP-Header. The combination of auth_host and auth_port should be used to validate and ask new token from Keystone. auth_host should point to the admin url of keystone.

At the moment, Heat can't be install in an environment where the publicURL (the one referenced by auth_uri) is not reachable from the Heat engine. Other Openstack projects deals fine with this network topology.

This bug report relates to : https://bugs.launchpad.net/heat/+bug/1259968

[1] http://lists.openstack.org/pipermail/openstack-dev/2013-August/012886.html
[2] https://review.openstack.org/#/c/39796/
[3] https://github.com/openstack/python-keystoneclient/blob/master/keystoneclient/middleware/auth_token.py#L191

Revision history for this message
Steven Hardy (shardy) wrote :

So there appear to be two issues here:

1. For both the core and contrib v2 keystoneclient wrapper, we use auth_uri for the auth_url passed into keystoneclient.

Arguably, as reported, this should instead use the auth_host/auth_port to connect to the admin endpoint for connections made as an admin (e.g when not using auth_url from the context, i.e using X-Auth-Url in standalone mode)

2. In the core keystoneclient wrapper, we override what is in the keystone catalog by passing auth_url and endpoint

This was to enable reliable access to v3 keystone functionality in environments where the catalog endpoints were versioned as v2.0. It may be that we can remove this now, as I believe work has been completed in keystoneclient to enable version discovery - hopefully this means we can move to just passing the auth_url and letting the client work out the endpoint from the catalog.

In both cases, my main concern is breaking existing functionality, we'll have to do some testing and figure out what works.

An alternative, simpler/less-risky solution is to provide a heat-specific configuration for admin connections to keystone. Arguably we're doing the wrong thing by reusing API configuration in the engine so perhaps having this (optionally) decoupled in the config file makes sense.

Changed in heat:
status: New → Triaged
importance: Undecided → Medium
milestone: none → juno-1
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-1 → juno-2
Steven Hardy (shardy)
Changed in heat:
milestone: juno-2 → juno-3
Thierry Carrez (ttx)
Changed in heat:
milestone: juno-3 → juno-rc1
Zane Bitter (zaneb)
Changed in heat:
assignee: nobody → Steven Hardy (shardy)
Zane Bitter (zaneb)
Changed in heat:
milestone: juno-rc1 → kilo-1
Angus Salkeld (asalkeld)
Changed in heat:
milestone: kilo-1 → kilo-2
Revision history for this message
Jamie Lennox (jamielennox) wrote :

This problem I think can be generalized to heat (or anyone) should not use, rely on or in any way touch the configuration of auth_token middleware. This configuration is used specifically for token validation and heat shouldn't need to be reacting when the config options of auth_token middleware changes.

Steven Hardy (shardy)
Changed in heat:
milestone: kilo-2 → kilo-3
Revision history for this message
Adam Gandelman (gandelman-a) wrote :

@Jamie-

I discovered devstack is currently setting incorrect/obsolete auth_token options [1] The issue cropped up for me in Ironic, which contains a code path that pulls out specific config options from the auth_token section [2] Your comments here make it sound like that is an Ironic bug, and that it should be creating a client in some other way. Thoughts?

[1] https://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/keystone#n425 (ie, username instead of admin_user)
[2] https://git.openstack.org/cgit/openstack/ironic/tree/ironic/common/keystone.py#n63

Changed in sahara:
status: New → Confirmed
importance: Undecided → Medium
milestone: none → liberty-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to sahara (master)

Reviewed: https://review.openstack.org/164274
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=a8e8963b6868948c875545e5bfdeeb6e3511a1ae
Submitter: Jenkins
Branch: master

commit a8e8963b6868948c875545e5bfdeeb6e3511a1ae
Author: Sergey Reshetnyak <email address hidden>
Date: Fri Mar 13 20:19:03 2015 +0300

    Changing method for verifying existence of cinder

    Sahara uses keystone admin client for checking existence of cinder.
    This patch change method for checking existence of cinder

    Change-Id: I082c2392742476eb7fa1e74d2443d10c89d6f707
    Partial-bug: #1300246

Angus Salkeld (asalkeld)
Changed in heat:
milestone: kilo-3 → kilo-rc1
Angus Salkeld (asalkeld)
tags: added: kilo-rc-potential
Changed in heat:
milestone: kilo-rc1 → next
Angus Salkeld (asalkeld)
tags: removed: kilo-rc-potential
Changed in sahara:
milestone: liberty-1 → liberty-2
Thierry Carrez (ttx)
Changed in sahara:
milestone: liberty-2 → liberty-3
Changed in sahara:
milestone: liberty-3 → liberty-rc1
Changed in sahara:
milestone: liberty-rc1 → next
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on sahara (master)

Change abandoned by Sergey Lukjanov (<email address hidden>) on branch: master
Review: https://review.openstack.org/163873
Reason: This path is older then 3 month, abandoned

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

This issue continues to cause major headaches during installation guide development and probably any deployment of heat, especially after keystone middleware deprecated the identity_uri, admin_tenant_name, admin_user, and admin_password options. Reuse of options aside, heat should at least support the following options (example values shown) in the [keystone_authtoken] section:

auth_uri = http://controller:5000 (note lack of version here)
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = heat
password = heatpass

Revision history for this message
Steven Hardy (shardy) wrote :

@Matt Kassawara: Thanks for raising this, however please can you clarify your comment #6 - AIUI heat should not be using the keystone_authtoken config, as mentioned in comment #2, so we now support a separate "clients_keystone" section instead, which is completely independent of the keystone_authtoken section.

To avoid breaking existing users we still support a fallback mode to use keystone_authtoken, but it is no longer required.

However you appear to be requesting action related to the keystone_authtoken config, which, AIUI, should not be required given that section is exclusively for use by keystonemiddleware?

If you can further clarify exactly what major headaches heat is causing I'll be glad to revive this and work to resolve them, but I was under the impression this had been pretty much indirectly fixed via other bugs.

https://github.com/openstack/heat/blob/master/heat/common/config.py#L299

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Oops... probably should provide additional comments here rather than our e-mail thread.

The example configuration file from "tox -e genconfig" only includes the auth_uri option in the [clients_keystone] section. Only using this option in my environment yields a 400 error from heat.common.wsgi with the following message:

BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.

I can resolve this issue only by adding the admin_tenant_name, admin_user, and admin_password options in the [keystone_authtoken] section. However, keystone middleware deprecated these options in Kilo and no other services require them.

Additionally, the heat.conf file in the heat gate jobs does not implement [clients_keystone] and continues to use the auth_uri and deprecated admin_* options in the [keystone_authtoken] section.

Steven Hardy (shardy)
Changed in heat:
milestone: next → mitaka-1
importance: Medium → High
Revision history for this message
Steven Hardy (shardy) wrote :

@Matt: Thanks for the update.

I'll have to test to check, but I think that error is coming from using the auth_token user as the trustee for deferred authentication (keystone trusts).

That is also fixed, by the addition of a new "trustee" config group:

https://github.com/openstack/heat/blob/master/heat/common/context.py#L156

So, I think, the main issue is devstack and any docs need updating to reflect this.

I've targetted this at M1 so we can ensure the devstack default config switches over to avoid using any deprecated interfaces, which I hope will address your concerns.

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Why doesn't the trustee group and options appear in the configuration file?

Revision history for this message
Steven Hardy (shardy) wrote :

> Why doesn't the trustee group and options appear in the configuration file?

I think the answer is because the new section was added via a keystoneclient auth plugin, which has it's own config registration handle.

I need to sync up with those involved with fixing https://bugs.launchpad.net/heat/+bug/1446918, then we can work out how to fix up the config generation to include the new section.

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

The keystone client or middleware?

Revision history for this message
Steven Hardy (shardy) wrote :

https://github.com/openstack/heat/blob/master/heat/common/context.py#L38

The new group is added via keystoneclient.auth - I'm not familiar with why that's done like that, and neither is miguelgrinberg the co-author of the patch.

I'll check with Jamie Lennox, since he invented the auth-plugin mechanism in keystoneclient IIRC, and wrote much of the heat code porting to use them. He'll either be able to clarify how that's supposed to work with genconfig, or confirm that we can convert to directly registering directly via cfg.CONF.

Changed in heat:
milestone: mitaka-1 → mitaka-2
Revision history for this message
Steven Hardy (shardy) wrote :

Ok, sorry for the delay on this - I'm now working on fixing the sample config generation so it includes the "trustee" section, and fixing up devstack so it uses that by default.

Changed in heat:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/254145

Revision history for this message
Steven Hardy (shardy) wrote :

I've proposed a patch which adds the keystoneclient options for the "trustee" section to the output from tox -e genconfig.

In summary, you can maintain the previous behavior, which is to delegate to the heat service user defined in the auth_token section by adding this to your heat.conf

[trustee]
auth_plugin = v3password
auth_url = http://<keystone hostname or IP>:5000/v3
username = heat
password = password
user_domain_id = default

Note, this can be maintained independently of any auth_token section, so I believe when that lands, we can close this bug?

I'll also work on some better docs describing how this all works, and update devstack to match this configuration by default.

Revision history for this message
Matt Kassawara (ionosphere80) wrote :

Can [trustee] use" auth_plugin = password" with "auth_url" lacking a version similar to how the installation guide uses the password plug-in for all services? For example, glance [1]. Also, is "auth_uri" necessary? Here's what the installation guide currently uses for the [trustee] section [2]:

[trustee]
auth_uri = http://controller:5000
auth_url = http://controller:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = heat
password = HEAT_PASS

[1] http://docs.openstack.org/liberty/install-guide-ubuntu/glance-install.html#install-and-configure-components
[2] http://docs.openstack.org/liberty/install-guide-ubuntu/heat-install.html#install-and-configure-components

Revision history for this message
Steven Hardy (shardy) wrote :

@Matt:

So, I did some more testing, and it appears that using an unversioned auth_url, and the generic "password" auth_plugin does the right thing:

[trustee]
auth_plugin = password
auth_url = http://192.168.1.14:5000
username = heat
password = foobar
user_domain_id = default

I do not believe the auth_uri entry is necessary at all - I've not used it for testing and I can't see it referenced anywhere in keystoneclient/auth/* (e.g the client plugins which consume this section)

I believe the project_name and project_domain_id in the docs is wrong - in fact I'm pretty sure including those will break completely with the default setting of deferred_auth_method=trusts, because we want the client to get a trust-scoped token, and it can't be scoped to both a trust and the project.

Testing proves that - keystone gives "Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped" if you try the above.

So, we've got some docs errors to work out - thanks for highlighting them.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-manuals (master)

Fix proposed to branch: master
Review: https://review.openstack.org/254327

Changed in openstack-manuals:
assignee: nobody → Steven Hardy (shardy)
status: New → In Progress
Revision history for this message
Steven Hardy (shardy) wrote :

Matt: Docs patch proposed which I believe resolves the errors you mentioned - please provide feedback on the review if you're happy it addresses your concerns, thanks!

Revision history for this message
Steven Hardy (shardy) wrote :

It's somewhat hard to know the best approach in terms of sample config generation - from the heat perspective all we define is a "trustee" config group, and the required contents of that are entirely dependent on keystoneclient, and which auth_plugin is selected.

So, arguably we should only be documenting the "auth_plugin" and "auth_section" options which are common to all plugins, only that makes the sample config less self documenting than what I've proposed in https://review.openstack.org/#/c/254145/, e.g there we accept that the only known plugin that works in this case is a password plugin that supports trusts, but in theory other plugins could work provided trust scoping is possible (we pass trust_id internally)

Changed in openstack-manuals:
assignee: Steven Hardy (shardy) → Matt Kassawara (ionosphere80)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-manuals (master)

Reviewed: https://review.openstack.org/254327
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=98ef59a59973954f5396df4898ffc1d1df98b9c2
Submitter: Jenkins
Branch: master

commit 98ef59a59973954f5396df4898ffc1d1df98b9c2
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 17:33:42 2015 +0000

    [install] Fix heat trustee configuration

    The current example config is broken with the default deferred
    authentication of trusts - the project_domain_id and project_name
    cannot be specified in this section or keystone will throw a
    "cannot be scoped to multiple targets" error when we attempt to get
    a token scoped to a trust.

    Also, the auth_uri is unused by the keystoneclient password auth
    plugins, so remove it (it has been a source of confusion in the
    referenced bug).

    Change-Id: I6305be3c693993de4d16d82fd6de936e92c437c5
    Closes-Bug: #1300246
    Backport: liberty

Changed in openstack-manuals:
status: In Progress → Fix Released
Revision history for this message
Steven Hardy (shardy) wrote :

I pushed a patch to devstack which aims to align the default devstack heat.conf with the changes discussed here and in the docs:

https://review.openstack.org/254755

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/254145
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=67f12e5e24eb9ec06e609f4399008b9d704128a1
Submitter: Jenkins
Branch: master

commit 67f12e5e24eb9ec06e609f4399008b9d704128a1
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 11:20:25 2015 +0000

    Add trustee config group to sample config

    Some time ago, we added support for a new "trustee" config section,
    aimed at allowing independent configuration of the credentials used
    to obtain trust-scoped tokens for deferred authentication. One of the
    main reasons for this was to avoid incorrectly using the keystone
    auth_token section, and as such a deprecation warning was added when
    we fall back to using that config section for heat.

    Unfortunately we didn't capture this new section in the sample config
    because it's registered via keystoneclient.auth, so this adds support
    for this section to the sample config generated via tox -e genconfig,
    and adds some notes clarifying usage to the auth_plugin option.

    To move to the new config syntax, but maintain the current behavior,
    which is to delegate to the heat service user, you can add this section
    to your heat.conf

    [trustee]
    auth_plugin = password
    auth_url = http://<keystone hostname or IP>:35357
    username = heat
    password = password
    user_domain_id = default

    The generated config documents many more options, all of those supported
    by the keystoneclient v3 Password auth plugin, but these are the minimum
    to enable delegation to the heat service user in the default domain.

    In new deployments this could be set to some other user (such as one created
    in the heat domain), but note that the trustee should not be changed for
    existing deployments where stacks exist, as the trust stored inside heat
    defines a relationship between the stack owner (trustor) and a specific
    trustee (which will be the heat service user if the deployment is using
    the deprecated path that steals credentials from keystone auth_token).

    Change-Id: I30aeb765a2246ce54b10972ae7187655d85cde1f
    Partial-Bug: #1300246

Revision history for this message
Liyingjun (liyingjun) wrote :

When deploying with devstack/keystone v3, the configuration is in keystone_authtoken section is still using admin_tenant_name,admin_password,admin_user, and this causes error: Identity response: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}

Changed in devstack:
assignee: nobody → Liyingjun (liyingjun)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to devstack (master)

Fix proposed to branch: master
Review: https://review.openstack.org/261398

Changed in devstack:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to devstack (master)

Reviewed: https://review.openstack.org/261398
Committed: https://git.openstack.org/cgit/openstack-dev/devstack/commit/?id=44ee7e307b49c5ee9e30ba380e36576189efdc34
Submitter: Jenkins
Branch: master

commit 44ee7e307b49c5ee9e30ba380e36576189efdc34
Author: liyingjun <email address hidden>
Date: Fri Dec 25 10:32:14 2015 +0800

    Fix heat config when using keystone v3

    Use configure_auth_token_middleware to generate keystone_authtoken
    section configration.

    Change-Id: I87e0e60afb958683add2aff2552d26cbf8c9e374
    Closes-bug: #1300246

Changed in devstack:
status: In Progress → Fix Released
Revision history for this message
Steven Hardy (shardy) wrote :

The heat patch landed, and devstack/docs are updated, so I think the heat part of this can be marked fixed - please let me know if you think there is anything further we need to do.

Changed in heat:
status: In Progress → Fix Committed
Revision history for this message
Matt Kassawara (ionosphere80) wrote :

The patch works for me.

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Change status to Fix Released (now it's a correct status for fixed bug)

Changed in heat:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to openstack-manuals (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/270598

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to openstack-manuals (stable/liberty)

Reviewed: https://review.openstack.org/270598
Committed: https://git.openstack.org/cgit/openstack/openstack-manuals/commit/?id=5a3618d4f51a64cc9ef16d9365aa7190b3f5914e
Submitter: Jenkins
Branch: stable/liberty

commit 5a3618d4f51a64cc9ef16d9365aa7190b3f5914e
Author: Steven Hardy <email address hidden>
Date: Mon Dec 7 17:33:42 2015 +0000

    [install] Fix heat trustee configuration

    The current example config is broken with the default deferred
    authentication of trusts - the project_domain_id and project_name
    cannot be specified in this section or keystone will throw a
    "cannot be scoped to multiple targets" error when we attempt to get
    a token scoped to a trust.

    Also, the auth_uri is unused by the keystoneclient password auth
    plugins, so remove it (it has been a source of confusion in the
    referenced bug).

    Change-Id: I6305be3c693993de4d16d82fd6de936e92c437c5
    Closes-Bug: #1300246
    Backport: liberty
    (cherry picked from commit 98ef59a59973954f5396df4898ffc1d1df98b9c2)

tags: added: in-stable-liberty
tags: removed: in-stable-liberty
Changed in sahara:
milestone: next → none
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/openstack-manuals 15.0.0

This issue was fixed in the openstack/openstack-manuals 15.0.0 release.

Revision history for this message
Marianne Linhares Monteiro (mariannelinharesm) wrote :

Is this bug valid for Sahara?

Revision history for this message
Telles Mota Vidal Nóbrega (tellesmvn) wrote :

Jeremy Freudberg, does your fix with trustee fixes this issue as well?
https://review.openstack.org/#/c/524936/

Revision history for this message
Jeremy Freudberg (jfreud) wrote :

As of https://review.openstack.org/#/c/524936/ , Sahara does not rely on or in any way touch the configuration of auth_token middleware.

So, for Sahara, the bug is resolved.

Changed in sahara:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers