Ironic API fails when keystone /v2.0 pipeline is disabled

[Davanum Srinivas (DIMS)]

Looks like Ironic cannot function with just Keystone /v3 API. Found this when i am trying to completely disable keystone /v2 usage in devstack (https://review.openstack.org/#/c/221300/)

Stack trace in ir-api log:
2015-09-11 14:02:34.244 11471 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2015-09-11 14:02:34.250 11471 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "The resource could not be found.", "code": 404, "title": "Not Found"}}
2015-09-11 14:02:34.250 11471 DEBUG keystonemiddleware.auth_token [-] Token validation failure. _fetch_token /usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py:843
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token Traceback (most recent call last):
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/__init__.py", line 831, in _fetch_token
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token data = self._identity_server.verify_token(token)
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token File "/usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/_identity.py", line 232, in verify_token
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token raise exc.InvalidToken(msg)
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token InvalidToken: Failed to fetch token data from identity server
2015-09-11 14:02:34.250 11471 ERROR keystonemiddleware.auth_token
2015-09-11 14:02:34.252 11471 DEBUG keystonemiddleware.auth_token [-] Marking token as unauthorized in cache store_invalid /usr/local/lib/python2.7/dist-packages/keystonemiddleware/auth_token/_cache.py:175

(picked from http://logs.openstack.org/00/221300/26/check/gate-tempest-dsvm-ironic-pxe_ssh/3558074//logs/screen-ir-api.txt.gz#_2015-09-11_14_02_34_244)

Evidence from keystone-access logs:
127.0.0.1 - - [11/Sep/2015:14:02:34 +0000] "POST /v2.0/tokens HTTP/1.1" 404 93 "-" "ironic/4.1.1.dev17 keystonemiddleware.auth_token/2.2.0" 1521(us)
127.0.0.1 - - [11/Sep/2015:14:02:34 +0000] "POST /v3/auth/tokens HTTP/1.1" 201 6242 "-" "python-keystoneclient" 68055(us)
127.0.0.1 - - [11/Sep/2015:14:02:34 +0000] "POST /v2.0/tokens HTTP/1.1" 404 93 "-" "ironic/4.1.1.dev17 keystonemiddleware.auth_token/2.2.0" 1198(us)

(picked from http://logs.openstack.org/00/221300/26/check/gate-tempest-dsvm-ironic-pxe_ssh/3558074//logs/apache/keystone_access.txt.gz)

[Pavlo Shchelokovskyy (pshchelo)]

dims, can you please be more specific on how to reproduce this?

unfortunately I can not install devstack with V3 only (various places are failing, seems that handling of ENABLE_IDENTITY_V2=False is not supported in DevStack for most of the services)

but using a standard install with both V2 and V3 when I do the following

$ export IDENTITY_API_VERSION=3
$ . /opt/stack/devstack/openrc admin admin
$ ironic --os-user-domain-name default --os-project-domain-name default driver-list

I see tokens being validated against V3 only

[Davanum Srinivas (DIMS) (dims-v)]

Pavlo,

I believe you should see it with just this change:
https://review.openstack.org/#/c/221300/27/lib/ironic,cm

If not, Apply the latest changeset in addition to ^^^ as i had removed this line in the last version of the changeset:
https://review.openstack.org/#/c/221300

Thanks,
Dims

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/236982
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f9ea26ebf33118cfc179cc183588df2a829db4b6
Submitter: Jenkins
Branch: master

commit f9ea26ebf33118cfc179cc183588df2a829db4b6
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Wed Mar 23 17:54:59 2016 +0200

    Migrate to using keystoneauth Sessions

    We currently construct Keystone client objects directly, which
    is no longer the preferred way. Instead, we should be using Sessions
    which allows use of different auth plugins. This change attempts to
    migrate our Keystone usage to this model.

    Additionally, we currently rely on the imported keystonemiddleware
    auth_token's configuration for all of the Keystone credentials used
    by the Ironic service user. This is bad, as that config is internal
    to that library and may change at any time. Also, the service user
    may be using different credentials than the token validator.

    This refactors the keystone module to use Sessions.
    It attempts to provide some backward compat for users
    who have not yet updated their config,
    by falling back to the authtoken config section when required.

    Operators impact:

    - Authentification parameters for each service now should specified in
      the corresponding config section for this service ([glance], [neutron]
      [swift], [inspector]).
      This includes providing both Keystone session-related options
      (timeout, SSL-related ones) and authentification options
      (`auth_type`, `auth_url` and proper options for the auth plugin).

    - New config section `service_catalog` for Ironic service user
      credentials, used to resolve Ironic API URL from Keystone catalog.

    - If loading from the service config section fails, an attempt is made
      to use respective options from [keystone_authtoken] section as a
      fall-back for backward compatibility.

    Implementation details:

    - using keystoneauth1 library instead of keystoneclient

    - For each service the keystone session is created only once and is
      reused further. This lowers the number of authentification requests
      made to Keystone but implies that only auth plugins that can
      re-authentificate themselves can be used (so no *Token plugins).

    This patch does not update the DevStack plugin, in order to test
    backwards compatibility with old config options.
    DevStack plugin will be modified in a subsequent patch.

    Change-Id: I166eebefc1e1335a1a7b632149cf6441512e9d5e
    Closes-Bug: #1422632
    Related-Bug: #1418341
    Related-Bug: #1494776
    Co-Authored-By: Adam Gandelman <email address hidden>

[Pavlo Shchelokovskyy (pshchelo)]

Dims, could you please re-test it as we've moved Ironic to full keystoneauth support.

[Ruby Loo (rloo)]

Just ping'd Dims, he's really busy and won't have time to look at this for a few weeks.

[Jay Faulkner (jason-oldos)]

Ironic now works with Keystone v3 in the gate. As far as I can tell, this bug is resolved. Please re-open if it's not.