Novaclient does not support v3 Keystone API

[Bogdan]

I am trying to setup Keystone Federation to a 3rd party Identity Provider via SAML2 and SSO for Horizon (all this on Kilo) and I believe I did these but as a consequence I now need to configure all OpenStack services to use v3 Keystone API as well as SSL.
It seems that the novaclient does not support the v3 of Keystone API thus many operations in Nova and Neutron fail.
How can I enable v3 in my Kilo stack? Is it possible?
Thanks in advance!

[melanie witt (melwitt)]

It's true, novaclient doesn't yet support v3 keystone auth, that is, it doesn't support the request/response format for the keystone v3 tokens call. The openstackclient, however, does have support [1]. The openstackclient uses novaclient underneath for calling nova apis, so as long as a call is made through novaclient with an already obtained token, it works (openstackclient code obtains the token and passes it through novaclient code). Similarly, nova should work (albeit unaware of any hierarchical tenancy concepts) as long as the api is called with the v3 token. That means usual calls work but you can't assign hierarchical project quota, for example.

Hope this helps.

[1] http://docs.openstack.org/developer/python-openstackclient/authentication.html#authenticating-using-identity-server-api-v3

[Bogdan (bogdan-vatkov)]

Thanks Melanie!

From what you explained I get it that working with openstackclient is fine but how about Horizon operation?

Now whenever I browse the /project/ url of Horizon for example I get the "cannot access" error in Horizon and in the nova.log I see

2015-12-03 17:45:05.815 17457 ERROR nova.compute.manager [req-bc22b21c-cf6a-4dbd-97f2-772301d65a80 – – – – -] [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] An error occurred while refreshing the network cache.
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] Traceback (most recent call last):
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 5596, in _heal_instance_info_cache
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] self._get_instance_nw_info(context, instance)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/compute/manager.py”, line 1360, in _get_instance_nw_info
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return self.network_api.get_instance_nw_info(context, instance)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 747, in get_instance_nw_info
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] preexisting_port_ids)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 763, in _get_instance_nw_info
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] preexisting_port_ids)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 1515, in _build_network_info_model
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] client = get_client(context, admin=True)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py”, line 205, in get_client
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] auth_token = _ADMIN_AUTH.get_token(_SESSION)
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] File “/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py”, line 104, in get_token
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f882843b52] return self.get_access(session).auth_token
2015-12-03 17:45:05.815 17457 TRACE nova.compute.manager [instance: eaec87f9-6e7c-46c8-8936-39f...

[melanie witt (melwitt)]

Horizon uses openstackclient, so it's compatible with keystone v3.

You just won't be able to use novaclient cli with keystone v3 -- use openstackclient in that case.

Hm, that might be the way to go (leave nova configured with v2). Sorry I don't know more about it. Nova hasn't been updated yet with keystone-v3-specific functionality. But when nova is provided with a v3 token, it can pass that around without knowledge of anything inside it, and as long as keystone v2 will honor the token, it should work fine with nova configured with v2. You might have to try it out a bit to see how it works out. We don't yet have CI jobs set up with nova to run alongside keystone v3.

[Lin Hua Cheng (lin-hua-cheng)]

horizon does not use python-openstackclient, it actually talks to novaclient directly.

Horizon passes the token directly to novaclient, so if you're on SSO, you'll end up passing v3 token to Nova.

Most operators run keystone v2 and v3 at the same time, it should work fine for you.

Something to note, v3 and v2 tokens works on v3 endpoint, while v2 endpoint will only work with v2 token. tokens are backward compatible but not forward compatible.

[John Garbutt (johngarbutt)]

Seems like this affects cinder. when they get a v3 token passed in, they can't do the assisted snapshot call via python-novaclient.

[Jordan Pittier (jordan-pittier)]

python-novaclient is used by Cinder for its assisted_volume_snapshots feature. See [1]

It means that Cinder is broken in a Keystone v3 environment because of this bug.

The code here [2] assumes that if auth_version if not v2.0 then "v2.0" will be appended to the auth_url. So if auth_url is "http://xx:35357/v3" then python-novaclient will connect to "http://xx:35357/v3/v2.0".

There's a tentative patch here : https://review.openstack.org/#/c/264764

This is problematic because https://review.openstack.org/#/c/271508/ (merged 4 days ago) set Keystone v3 as the default Keystone API version. As glusterfs, quobyte, scality and SMB drivers rely on assisted_volume_snapshots, their CI are broken (note: there are known workarounds).

[1] https://github.com/openstack/cinder/blob/55744deafb3fd8b4c7ba093626836d5a92255996/cinder/compute/nova.py#L137
[2] https://github.com/openstack/python-novaclient/blob/209f17224224e000a7e2c1ed3e698d7f7cb36864/novaclient/client.py#L558-L586

[Jordan Pittier (jordan-pittier)]

For reference: the workaround for cinder is to hardcode the auth_url using the Cinder config flag "os_privileged_user_auth_url" See [1] for details.

[1] https://github.com/openstack/cinder/blob/55744deafb3fd8b4c7ba093626836d5a92255996/cinder/compute/nova.py#L114

[Jordan Pittier (jordan-pittier)]

This is also a problem for Heat. They are currently going to workaround with https://review.openstack.org/#/c/274159 but the `replace("v3", "v2.0")` is obviously suboptimal.

[Mohammed Ashraf (mohammed-asharaf)]

Would like to work on this issue. Hence looking for following details..

1. Does nova-client support keystone v3 authentication. If not yet, then probably this work has to go thru blueprint process?
2. If it's supposed to be supported, then it could be broken?
3. If it's supported, is there a steps to reproduce this issue.

Thanks in Adv

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/289369
Committed: https://git.openstack.org/cgit/openstack/python-novaclient/commit/?id=e458fab8efdaab327c8f07a0771e4d3571580a41
Submitter: Jenkins
Branch: master

commit e458fab8efdaab327c8f07a0771e4d3571580a41
Author: Andrey Kurilin <email address hidden>
Date: Mon Mar 7 16:44:24 2016 +0200

    Add default values for domain related options

    Keystone V3 requires that the options os_user_domain_id/os_user_domain_name
    and os_project_domain_id/os_project_domain_name be set. These options do not
    have default values and novaclient will fail if they are not set by user.

    This patch adds default values for os_project_domain_id and
    os_user_domain_id in case of Keystone V3 env.
    This is consistent with what the OpenStack CLI does.

    Also, two functional tests are added to check authentication via
    Keystone V2 and V3.

    Partial-Bug: #1522402

    Change-Id: Ieb41a7fbf24107f7250ef34218604fba96531195

[melanie witt (melwitt)]

I've learned more about how novaclient works since my last comments. Novaclient works with keystone v3 as long as the keystoneauth session API is used (where keystoneclient library takes care of knowing the token request/response format, etc).

Instructions for that are here for the Python API:

http://docs.openstack.org/developer/python-novaclient/api.html#usage

Note that you must create a Session object and pass it to novaclient.Client(session=<session object>) when creating the novaclient object. If you don't, novaclient will fall back on its own code for keystone auth and will NOT work with keystone v3.

For the novaclient CLI, it will work with keystone v3 as long as you provide the additional keystone v3 specific arguments on the command line [1]. The patch in comment #10 provides defaults for two of the arguments so things can work in the absence of specifying the arguments on the command line, provided that "default" is okay for you. The patch is still on master and will be included in the next novaclient release.

[1] http://docs.openstack.org/developer/python-openstackclient/authentication.html#authenticating-using-identity-server-api-v3

[melanie witt (melwitt)]

Also important to note, with the novaclient CLI, the keystoneauth session API usage will be bypassed if any of these is true:

- an auth plugin other than keystone is being used
- token caching is enabled (OS_CACHE or --os-cache)
- a bypass url is being used
- volume_service_type is being specified i.e. --volume-service-type

and will NOT work with keystone v3.

[Sean McGinnis (sean-mcginnis)]

Ran into this too. I was able to get around it by following the keystoneauth1 session example here:

http://docs.openstack.org/developer/python-novaclient/api.html

After reading the fine print about what to do with keystone v3.

[Andriy Kurilin (andreykurilin)]

python API from novaclient 7.0.0 supports keystone V3 natively