"Failed to create trust" on pike

[Luigi Toscano]

puppet-sahara, after Ocata, sets the "new" keystone_authtoken values (related to Keystone v3, if I understand it correctly): https://review.openstack.org/#/c/441223/

But the configuration used by Sahara to setup trust (need also for cluster creation) uses the old variables names (, and then cluster creation fails ("Creating cluster failed for the following reason(s): Failed to create trust")

This affects all the installer based on puppet (including Packstack and TripleO)

Trying to fix this is not trivial because the keystone_authtoken values are kind of "private":
http://lists.openstack.org/pipermail/openstack-dev/2017-July/119944.html

See an attempt here:
https://review.openstack.org/#/c/485521/

Further testing showed that:

- apparently, even when the keystone_authtoken is imported, the unit test fails because auth_type is False by default so the new values (username, project_name, etc, vs the old admin_username, admin_tenant_name, etc) are not set:
http://git.openstack.org/cgit/openstack/keystonemiddleware/tree/keystonemiddleware/auth_token/__init__.py?id=efb1fb99d87f754a008877f2e2d391221cb43721#n885
Not sure where to properly change that value.

- even ignoring the unit test, apparently the key is not read also by the updated code; apparently an injection of the new parameters is needed. The problem is that it's not clear where (sahara/main.py?)

This should be fixed before Pike.

[Luigi Toscano (ltoscano)]

Interesting discovery that could lead to some results: when the two services (sahara-api and sahara-engine), the logs (in debug mode) shows that:

- sahara_api loads keystonemiddleware.auth_token, and contains the new values

INFO sahara.main [-] Sahara API started
DEBUG oslo.service.wsgi [-] Loading app sahara from /etc/sahara/api-paste.ini load_app /usr/lib/python2.7/site-packages/oslo_service/wsgi.py:352
INFO keystonemiddleware.auth_token [-] Starting Keystone auth_token middleware

- that snipped is not visible in the logs of sahara-engine, which does not load the new values.

So something is different in the loading code of the two services.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/485521
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=5abae32028c3998c067afd99ef7967b393a23071
Submitter: Jenkins
Branch: master

commit 5abae32028c3998c067afd99ef7967b393a23071
Author: Luigi Toscano <email address hidden>
Date: Thu Jul 20 13:19:58 2017 +0200

    Fully switch to keystone authtoken parameters

    The old v2 parameters are not set anymore by puppet-sahara:
    https://review.openstack.org/#/c/441223/
    and trust (which means cluster operations) is broken.

    Because puppet-sahara is used by TripleO and Packstack, we consider
    this a critical issue.

    We now switch to the "new" v3 parameters from keystone_authtoken, as
    incentivized by that puppet-sahara change.

    We no longer use the custom options admin_user_domain_name and
    admin_project_domain_name, as [keystone_authtoken] can provide them.

    Note 1: A workaround is needed to access some of the configs in
    [keystone_authtoken], as they are considered private for
    keystonemiddleware. In sahara-api, it would have been possible to
    grab these configs with only a slight bit of magic, as sahara-api
    is a keystonemiddleware-enabled WSGI application. However, with
    sahara-engine it is not as straightforward, since keystonemiddleware
    is not integrated there. Therefore, to access these private configs
    we use a very sneaky workaround inspired by [0]. This should be
    removed in Queens: we should add a separate, non-private
    [clients_keystone] section in sahara.conf. That is the standard way to
    grab service user credentials when excluded from access to
    [keystone_authtoken]. Unfortunately we could not have done that in Pike
    as it was too late to have a new puppet-sahara release.

    Note 2: tools/get_auth_token.py was not changed as it probably
    requires other changes to work with Identity v3.

    [0] Ibbc738ee4c90392af47f1b6d69aee3c8dbbf3c17

    Closes-Bug: #1709091
    Co-Authored-By: Jeremy Freudberg <email address hidden>

    Change-Id: I930e544b16f0871f5e8dc1a42aae34518ff25bcd

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara 7.0.0.0rc1 release candidate.