Heat not working with keystoneclient_v2

[Vijendar Komalla]

It seems Heat is not respecting 'keystone_backend' setting. I have set 'keystone_backend' to 'heat_keystoneclient_v2', but it is using keystoneclient_v3 (while debugging I noticed that, self.context.auth_plugin has v3 identity endpoint).
I am not sure if this change https://review.openstack.org/#/c/146368 introduced the behavior I am seeing.

[Jamie Lennox (jamielennox)]

Ok, so this is correct and not correct.

To see this you essentially need to use heat standalone:

Set:

[DEFAULT]
keystone_backend = heat_keystoneclient_v2.client.KeystoneClientV2

[paste_deploy]
flavor = standalone

Authentication will still work it will just use v3 to do it and not respect the v2.client setting.

This is because what i've done is to essentially separate the auth process from the CRUD process. If you are using any of the user or domain functions they will still operate on the keystone v2 API, they will just try and do it using a v3 token - which as far as I know will work just fine.

The error is: https://github.com/openstack/heat/blob/36eaea8208ab3e4b077f21b4c8eb95716296e9c6/heat/common/context.py#L194 where if a username and password are the only auth context items you have (so standalone without auth_token) then it constructs a v3.Password auth plugin - where in the case of keystoneclient_v2 it should construct a v2.Password.

I'll need to have a think as I don't see that there is a good way of handling this.

Is it reasonable to say that if you are using the auth_password middleware then you will always want to use v2 authentication?
Otherwise the only thing i can think of is to do a check on the client type, so if isinstance(kc_client, v2) v2.Password else v3.Password

[Jamie Lennox (jamielennox)]

Please ignore "Ok, so this is correct and not correct." I thought i had figured something else out and rewrote my comment. It was supposed to have been deleted.

[Steven Hardy (shardy)]

@jamielennox: Thanks for the response! Hopefully vijendar-komalla can confirm, but I think the use-case here is not standalone, RAX use the v2 plugin for heat in a non-standalone environment with their keystone-v2-compatible auth service.

Or at least that's my understanding of it.

So in this case, it's not that they'll be using auth_password, it's just that they need a way to direct v2-compatible requests (e.g not any of the domains or trusts stuff, which isn't implemented in the KeystoneClientV2 plugin) to their auth service.

It's still not that clear to me exactly what the fault is here, but given the limited detail in the initial report, I guess we'd need to attempt to reproduce by disabling the keystone v3 endpoint, then configuring heat to use KeystoneClientV2 and proving things still basically work.

Ideally, Vijendar will come back with some additional analysis, given that he has ready access to an environment where this issue is reproduceable :)

[Drago (dragorosson)]

I have been looking into this. The problem from our end is that the GET v2.0/tokens/{token id} endpoint (used to validate tokens, see http://developer.openstack.org/api-ref-identity-v2.html#admin-tokens) does not return the service catalog, so when the Heat API validates the token sent to it by the heatclient, it doesn't get a service catalog back. Whatever data the API gets back from validating the token is sent to the engine. The new auth_plugin in the engine expects this data to have the service catalog.

There is a mismatch between what the V2 API documentation says (linked above) and what is actually in the code. It can be seen here that upon validating a V2 token, the service catalog is returned: http://developer.openstack.org/api-ref-identity-v2.html#admin-tokens. I have reproduced this behavior using a devstack instance as well.

It seems that the best fix for this is to update the V2 documentation and add the service catalog to that endpoint on our end (i.e. not an OpenStack code problem).

[Steve Baker (steve-stevebaker)]

Heat should be able to fetch the service catalog if it is ever missing from the context, so I would prefer heat to not assume that a token always comes with catalog, and to separately fetch the catalog when required.

[Drago (dragorosson)]

My apologies, that second link was supposed to be https://github.com/openstack/keystone/blob/master/keystone/token/providers/common.py#L582-L589

[Jamie Lennox (jamielennox)]

Steve: The problem is that when i did the auth plugins patch i was trying to move as much auth as i could to the context object. IMO this is where it makes the most sense rather than client calling to another client.

As part of that i did:

https://github.com/openstack/heat/blob/master/heat/common/context.py#L176

and

https://github.com/openstack/heat/blob/master/heat/common/context.py#L196

which made complete sense when i though that the common/heat_keystoneclient was the only one that really mattered. What used to happen was that the clients would call out to the configured keystoneclient for this and get v2 or v3 auth rather than v3 always. So now when you configure the v2 client you do keystone CRUD over the v2 api, but auth is still v3.

I still believe the context is the best place to handle auth, however in the tests i've done having the context call the client to fetch an appropriate plugin is tricky as you can end up in a recursive loop. I still am hoping to get this fixed, it's just not as easy as anticipated.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/159554
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=2913f0ee1404d72ddbaa0d753601554fcbe217cd
Submitter: Jenkins
Branch: master

commit 2913f0ee1404d72ddbaa0d753601554fcbe217cd
Author: Drago Rosson <email address hidden>
Date: Mon Mar 2 21:31:05 2015 +0000

    Retrieve service catalog when it is missing

    The Heat API uses the keystone endpoint GET /{version}/tokens/{token id}
    to validate tokens. The specification for the v2.0 version of this
    endpoint omits the service catalog in its response (although the
    OpenStack implementation of Keystone v2 still includes it). The
    response is passed along to the Heat engine which uses it as part of its
    RequestContext. Heat expects the service catalog to be in this context,
    so this patch retrieves it if it is not there.

    closes-bug: #1420987
    Change-Id: I183b22931fa8c997883168038e260db3d54b5fe5