keystone_authtoken configuration error in ironic.conf from devstack

[chenglch]

the [keystone_authtoken] in ironic.conf define
[keystone_authtoken]
password = secretservice
username = ironic
for example
http://logs.openstack.org/09/153109/2/check/check-tempest-dsvm-ironic-pxe_ssh/79080a0/logs/etc/ironic/ironic.conf.txt.gz

but in ironic/common/keystone.py that is
 if token:
            return client.Client(token=token, auth_url=auth_url)
        else:
            return client.Client(username=CONF.keystone_authtoken.admin_user,
                         password=CONF.keystone_authtoken.admin_password,
                         tenant_name=CONF.keystone_authtoken.admin_tenant_name,
                         region_name=CONF.keystone.region_name,
                         auth_url=auth_url)
https://github.com/openstack/ironic/blob/master/ironic/common/keystone.py#L67

so when the token is out of time , ironic deploy will failed as the error of authorize in Keystone

2015-02-05 00:48:18.532 DEBUG ironic.common.states [-] Exiting old state 'deploying' in response to event 'fail' from (pid=19884) on_exit /opt/stack/ironic/ironic/common/states.py:128
2015-02-05 00:48:18.533 DEBUG ironic.common.states [-] Entering new state 'deploy failed' in response to event 'fail' from (pid=19884) on_enter /opt/stack/ironic/ironic/common/states.py:134
2015-02-05 00:48:18.589 WARNING ironic.conductor.manager [-] Error in deploy of node 08a45b46-f123-4f19-a10d-faff54c8342b: Could not authorize in Keystone: A username and password or token is required.
2015-02-05 00:48:18.631 DEBUG ironic.conductor.task_manager [-] Attempting to reserve node 2 from (pid=19884) reserve_node /opt/stack/ironic/ironic/conductor/task_manager.py:193
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in fire_timers
    timer()
  File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in __call__
    cb(*args, **kw)
  File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
    result = function(*args, **kwargs)
  File "/opt/stack/ironic/ironic/conductor/manager.py", line 1458, in do_node_deploy
    _("Failed to deploy. Error: %s"))
  File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 82, in __exit__
    six.reraise(self.type_, self.value, self.tb)
  File "/opt/stack/ironic/ironic/conductor/manager.py", line 1453, in do_node_deploy
    new_state = task.driver.deploy.deploy(task)
  File "/opt/stack/ironic/ironic/conductor/task_manager.py", line 128, in wrapper
    return f(*args, **kwargs)
  File "/opt/stack/ironic/ironic/drivers/modules/pxe.py", line 329, in deploy
    _create_token_file(task)
  File "/opt/stack/ironic/ironic/drivers/modules/pxe.py", line 258, in _create_token_file
    token = keystone.get_admin_auth_token()
  File "/opt/stack/ironic/ironic/common/keystone.py", line 129, in get_admin_auth_token
    ksclient = _get_ksclient()
  File "/opt/stack/ironic/ironic/common/keystone.py", line 76, in _get_ksclient
    ' %s') % err)

[Yuki Nishiwaki (uckey-1067)]

Hello chenglch.

this is not problem . because the following configuration is invalid.
>[keystone_authtoken]
>password = secretservice
>username = ironic

the correct configuration is bellow.
>[keystone_authtoken]
>admin_password = secretservice
>admin_user = ironic
>admin_tenant_name = service

But if the invalid configuration file is created by devstack , Please display your version of devstack.
At least I have confirmed that current master branch of devstack is working well.

[chenglch (chenglch)]

Hello Yuki Nishiwaki .

You can see the test configuration on CI, like link
https://review.openstack.org/#/c/146307/
which is upload yesterday.
The ironic.conf file that check-tempest-dsvm-ironic-pxe_ssh jenkins job use can be found in the link below.

http://logs.openstack.org/07/146307/5/check/check-tempest-dsvm-ironic-pxe_ssh/964a5da/logs/etc/ironic/ironic.conf.txt.gz

The keystone_authtoken section in that file use
[keystone_authtoken]
>signing_dir = /var/cache/ironic/api
>cafile = /opt/stack/data/ca-bundle.pem
>auth_uri = http://127.0.0.1:5000
>project_domain_id = default
>project_name = service
>user_domain_id = default
>password = secretservice
>username = ironic
>auth_url = http://127.0.0.1:35357
>auth_plugin = password

rather than the correct configuration you mentioned.
>[keystone_authtoken]
>admin_password = secretservice
>admin_user = ironic
>admin_tenant_name = service

I think CI always use devstack-gate to get the latest devstack , that's very strange.

[Yuki Nishiwaki (uckey-1067)]

Sorry for making avtivity log dirty.

I realized that the version of devstack I used is old . And after I updated to current version , I also confirmed this bug is caused.

But , the reason of this problem is following commit of devstack .
--
Merge: 2d7c346 78b7726
Author: Jenkins <email address hidden>
Date: Mon Jan 26 14:06:15 2015 +0000

    Merge "Configure auth_token middleware by auth plugin"

commit 905275ec9f51614911fbb915f8bba9896b3ab684
--

And , the commit merged by above log is following ( <- sorry for poor English )

---
Author: Jamie Lennox <email address hidden>
Date: Fri Dec 19 12:56:01 2014 +1000

    Configure auth_token middleware by auth plugin

    As of release 1.3 auth_token middleware can be configured to use any
    authentication plugin. This allows us to move to the more generic
    password mechanism which will default to using keystone v3 if available.

    This will allow in future revisions to move the devstack service users
    out of the default domain.

    Work will need to be done in heat to remove it's dependency on the
    (supposed to be private) keystone_authtoken CONF values.

    Change-Id: Ieac26806bd420aa08fc79bbc6a11eb6a1c15c7df

diff --git a/lib/keystone b/lib/keystone
index 1599fa5..9c15688 100644
--- a/lib/keystone
+++ b/lib/keystone
@@ -407,15 +407,6 @@ function create_keystone_accounts {
     fi
 }

-# Configure the API version for the OpenStack projects.
-# configure_API_version conf_file version [section]
-function configure_API_version {
- local conf_file=$1
- local api_version=$2
- local section=${3:-keystone_authtoken}
- iniset $conf_file $section auth_uri $KEYSTONE_SERVICE_PROTOCOL://$KEYSTONE_SERVICE_HOST:$KEYSTONE_SERVICE_PORT/v$api_version
-}
-
 # Configure the service to use the auth token middleware.
 #
 # configure_auth_token_middleware conf_file admin_user signing_dir [section]
@@ -429,15 +420,16 @@ function configure_auth_token_middleware {
     local signing_dir=$3
     local section=${4:-keystone_authtoken}

- iniset $conf_file $section auth_host $KEYSTONE_AUTH_HOST
- iniset $conf_file $section auth_port $KEYSTONE_AUTH_PORT
- iniset $conf_file $section auth_protocol $KEYSTONE_AUTH_PROTOCOL
- iniset $conf_file $section identity_uri $KEYSTONE_AUTH_URI
+ iniset $conf_file $section auth_plugin password
+ iniset $conf_file $section auth_url $KEYSTONE_AUTH_URI
+ iniset $conf_file $section username $admin_user
+ iniset $conf_file $section password $SERVICE_PASSWORD
+ iniset $conf_file $section user_domain_id default
+ iniset $conf_file $section project_name $SERVICE_TENANT_NAME
+ iniset $conf_file $section project_domain_id default
+
+ iniset $conf_file $section auth_uri $KEYSTONE_SERVICE_URI
     iniset $conf_file $section cafile $SSL_BUNDLE_FILE
- configure_API_version $conf_file $IDENTITY_API_VERSION $section
- iniset $conf_file $section admin_tenant_name $SERVICE_TENANT_NAME
- iniset $conf_file $section admin_user $admin_user
- iniset $conf_file $section admin_password $SERVICE_PASSWORD
     iniset $conf_file $section signing_dir $signing_dir
 }

commit 157c84b8766ae67f6868f26505ce9a...

[Dmitry Tantsur (divius)]

So, correct ones according to http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html and https://github.com/openstack/keystonemiddleware/blob/b562b04ee5db309268716e0e1b8270f30bdf1a76/keystonemiddleware/auth_token.py#L645-L661 are ones with admin_ prefix. I have no idea, why in devstack it is different: https://github.com/openstack-dev/devstack/blob/9b215db569dcee2e9cd52c3336ba14b73ad05ab0/lib/keystone#L447-L449 but it might devstack problem. Ironic seems to follow the official documentation.

[Yuki Nishiwaki (uckey-1067)]

I think this problem is the bug of devstack , So I fixed it.
Please review it.

https://review.openstack.org/#/c/155360/

[Adam Gandelman (gandelman-a)]

This is a bug in ironic. We're creating a keystone client for the service user using the configured auth_token credentials, which are there for auth token validation, not inter service communication. This'll require a fix to both Ironic and devstack.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/156344

[Adam Gandelman (gandelman-a)]

So the bug in Ironic is that we dont want Ironic to be relying on the keystonemiddleware.auth_token's config settings, since those may change anytime. I'm trying to work out a patch to migrate Ironic away but provide some backward compat in the near future, but I'm having trouble figuring out the split brain between devstack's configure_auth_token_middleware()' and the config settings still setup in keystonemidddleware.auth_token. Do we expect keystonemiddleware's settings to change from admin_username to username in the near future, and the devstack difference is just in preparation?

[Yuki Nishiwaki (uckey-1067)]

The temporary work-around patch is merged to master branch.

[Dmitry Tantsur (divius)]

Adam, I'm assigning the Ironic part to you based on https://review.openstack.org/#/c/156344/

[OpenStack Infra (hudson-openstack)]

Change abandoned by Adam Gandelman (<email address hidden>) on branch: master
Review: https://review.openstack.org/156344

[vasanth (getvasanth)]

Sorry, that was wrongly marked as fixed. sorry about that. Can the admin please change the status to "In Progress"

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/236982

[OpenStack Infra (hudson-openstack)]

Change abandoned by Yuriy Zveryanskyy (<email address hidden>) on branch: master
Review: https://review.openstack.org/156344

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/238554

[OpenStack Infra (hudson-openstack)]

Change abandoned by Pavlo Shchelokovskyy (<email address hidden>) on branch: master
Review: https://review.openstack.org/238554
Reason: as Ironic has completely moved to DevStack plugin, I will re-propose this to Ironic tree

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/236982
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f9ea26ebf33118cfc179cc183588df2a829db4b6
Submitter: Jenkins
Branch: master

commit f9ea26ebf33118cfc179cc183588df2a829db4b6
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Wed Mar 23 17:54:59 2016 +0200

    Migrate to using keystoneauth Sessions

    We currently construct Keystone client objects directly, which
    is no longer the preferred way. Instead, we should be using Sessions
    which allows use of different auth plugins. This change attempts to
    migrate our Keystone usage to this model.

    Additionally, we currently rely on the imported keystonemiddleware
    auth_token's configuration for all of the Keystone credentials used
    by the Ironic service user. This is bad, as that config is internal
    to that library and may change at any time. Also, the service user
    may be using different credentials than the token validator.

    This refactors the keystone module to use Sessions.
    It attempts to provide some backward compat for users
    who have not yet updated their config,
    by falling back to the authtoken config section when required.

    Operators impact:

    - Authentification parameters for each service now should specified in
      the corresponding config section for this service ([glance], [neutron]
      [swift], [inspector]).
      This includes providing both Keystone session-related options
      (timeout, SSL-related ones) and authentification options
      (`auth_type`, `auth_url` and proper options for the auth plugin).

    - New config section `service_catalog` for Ironic service user
      credentials, used to resolve Ironic API URL from Keystone catalog.

    - If loading from the service config section fails, an attempt is made
      to use respective options from [keystone_authtoken] section as a
      fall-back for backward compatibility.

    Implementation details:

    - using keystoneauth1 library instead of keystoneclient

    - For each service the keystone session is created only once and is
      reused further. This lowers the number of authentification requests
      made to Keystone but implies that only auth plugins that can
      re-authentificate themselves can be used (so no *Token plugins).

    This patch does not update the DevStack plugin, in order to test
    backwards compatibility with old config options.
    DevStack plugin will be modified in a subsequent patch.

    Change-Id: I166eebefc1e1335a1a7b632149cf6441512e9d5e
    Closes-Bug: #1422632
    Related-Bug: #1418341
    Related-Bug: #1494776
    Co-Authored-By: Adam Gandelman <email address hidden>

[Pavlo Shchelokovskyy (pshchelo)]

I think this can be closed now, everything is merged in Ironic code and Ironic devstack plugin,
and now setting new auth_plugin based options in keystone_authtoken section should not break Ironic.