scn0001 multinode container: mistralclient api authorization failed

[Rafael Folco]

undercloud install fails with the following error:
APIException: Authorization failed: Cannot authenticate without an auth_url

http://logs.openstack.org/36/555336/5/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5f87f73/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-04-04_12_25_34

2018-04-04 12:25:34 | 2018-04-04 12:25:34,010 DEBUG: An exception occurred
2018-04-04 12:25:34 | Traceback (most recent call last):
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2291, in install
2018-04-04 12:25:34 | _post_config(instack_env, upgrade)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2008, in _post_config
2018-04-04 12:25:34 | _post_config_mistral(instack_env, mistral, swift)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1945, in _post_config_mistral
2018-04-04 12:25:34 | _create_logging_cron(mistral)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1911, in _create_logging_cron
2018-04-04 12:25:34 | pattern='0 * * * *'
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/v2/cron_triggers.py", line 55, in create
2018-04-04 12:25:34 | return self._create('/cron_triggers', data)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/base.py", line 97, in _create
2018-04-04 12:25:34 | self._raise_api_exception(ex.response)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/base.py", line 160, in _raise_api_exception
2018-04-04 12:25:34 | error_message=error_data)
2018-04-04 12:25:34 | APIException: Authorization failed: Cannot authenticate without an auth_url
2018-04-04 12:25:34 | 2018-04-04 12:25:34,013 ERROR:
2018-04-04 12:25:34 | #############################################################################
2018-04-04 12:25:34 | Undercloud install failed.
2018-04-04 12:25:34 |
2018-04-04 12:25:34 | Reason: Authorization failed: Cannot authenticate without an auth_url
2018-04-04 12:25:34 |
2018-04-04 12:25:34 | See the previous output for details about what went wrong. The full install
2018-04-04 12:25:34 | log can be found at /home/zuul/.instack/install-undercloud.log.
2018-04-04 12:25:34 |
2018-04-04 12:25:34 | #############################################################################
2018-04-04 12:25:34 |
2018-04-04 12:25:34 | Traceback (most recent call last):
2018-04-04 12:25:34 | File "<string>", line 1, in <module>
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2291, in install
2018-04-04 12:25:34 | _post_config(instack_env, upgrade)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 2008, in _post_config
2018-04-04 12:25:34 | _post_config_mistral(instack_env, mistral, swift)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1945, in _post_config_mistral
2018-04-04 12:25:34 | _create_logging_cron(mistral)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/instack_undercloud/undercloud.py", line 1911, in _create_logging_cron
2018-04-04 12:25:34 | pattern='0 * * * *'
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/v2/cron_triggers.py", line 55, in create
2018-04-04 12:25:34 | return self._create('/cron_triggers', data)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/base.py", line 97, in _create
2018-04-04 12:25:34 | self._raise_api_exception(ex.response)
2018-04-04 12:25:34 | File "/usr/lib/python2.7/site-packages/mistralclient/api/base.py", line 160, in _raise_api_exception
2018-04-04 12:25:34 | error_message=error_data)
2018-04-04 12:25:34 | mistralclient.api.base.APIException: Authorization failed: Cannot authenticate without an auth_url
2018-04-04 12:25:34 | Command '['instack-install-undercloud']' returned non-zero exit status 1

[Jose Luis Franco (jfrancoa)]

More info about the issue can be found in the mistral api log:

http://logs.openstack.org/74/548974/28/check/tripleo-ci-centos-7-undercloud-oooq/e994b66/logs/undercloud/var/log/mistral/api.log.txt.gz#_2018-04-04_12_07_03_388

2018-04-04 12:07:03.388 31578 ERROR wsme.api [req-80084090-9d73-47da-98a6-c4d6585f89cd 003ca7a913db410e885daacee3e9bc78 d1d9cc5ef7e542ba80cf5b8706a5c92d - default default] Server-side error: "Authorization failed: Cannot authenticate without an auth_url". Detail:
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/wsmeext/pecan.py", line 85, in callfunction
    result = f(self, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/mistral/api/controllers/v2/cron_trigger.py", line 78, in post
    workflow_id=values.get('workflow_id')

  File "/usr/lib/python2.7/site-packages/mistral/db/utils.py", line 88, in decorate
    return retry.call(_with_auth_context, auth_ctx, func, *args, **kw)

  File "/usr/lib/python2.7/site-packages/mistral/utils/rest_utils.py", line 247, in call
    return super(MistralRetrying, self).call(fn, *args, **kwargs)

  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 295, in call
    start_time=start_time)

  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 252, in iter
    return fut.result()

  File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
    return self.__get_result()

  File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 298, in call
    result = fn(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/mistral/db/utils.py", line 45, in _with_auth_context
    return func(*args, **kw)

  File "/usr/lib/python2.7/site-packages/mistral/services/triggers.py", line 130, in create_cron_trigger
    security.add_trust_id(trigger_parameters)

  File "/usr/lib/python2.7/site-packages/mistral/services/security.py", line 110, in add_trust_id
    trust = create_trust()

  File "/usr/lib/python2.7/site-packages/mistral/services/security.py", line 41, in create_trust
    client = keystone.client()

  File "/usr/lib/python2.7/site-packages/mistral/utils/openstack/keystone.py", line 42, in client
    auth_url=auth_url

  File "/usr/lib/python2.7/site-packages/keystoneclient/v3/client.py", line 250, in __init__
    self.authenticate()

  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 578, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)

  File "/usr/lib/python2.7/site-packages/keystoneclient/v3/client.py", line 336, in get_raw_token_from_identity_service
    _('Authorization failed: %s') % e)

AuthorizationFailure: Authorization failed: Cannot authenticate without an auth_url
: AuthorizationFailure: Authorization failed: Cannot authenticate without an auth_url

[Jose Luis Franco (jfrancoa)]

Seems that this recently added patch into puppet-keystone could have something to do with it:

 Deprecate auth_uri option

Option "auth_uri" from group "keystone_authtoken" is deprecated[1].
Use option "www_authenticate_uri" from group "keystone_authtoken".

[1]: https://review.openstack.org/#/c/508522/

Change-Id: I4c82a63baabd6b9304b302c97cd751a0103d8316
Closes-Bug: #1759098

https://github.com/openstack/puppet-keystone/commit/73d07c00f4e05cb14408f16560cbe20283955a1e

puppet-keystone version installed is now puppet-keystone-12.3.1-0.20180404092426.fa98723.el7.centos.noarch which does include that patch.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/558831

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/558827/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/558837

[Alex Schultz (alex-schultz)]

https://review.openstack.org/#/c/558827/ is merged

[OpenStack Infra (hudson-openstack)]

Change abandoned by Emilien Macchi (<email address hidden>) on branch: master
Review: https://review.openstack.org/558837
Reason: TO NOT RE-CHECK OR RE-APPROVE - CLEARING THE GATE NOW TO FIX A BLOCKER

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558837
Committed: https://git.openstack.org/cgit/openstack/instack-undercloud/commit/?id=be9bc031ff14cef29b2daa9d138272359f46e5d6
Submitter: Zuul
Branch: master

commit be9bc031ff14cef29b2daa9d138272359f46e5d6
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Apr 4 16:29:12 2018 +0200

    Replace deprecated auth_uri by www_authenticate_uri.

    auth_uri option has been depreacted in favor
    of www_authenticate_uri from group keystone_authtoken
    in puppet-keystone [0] and keystonemiddleware [1].

    This patch adds the new option www_authenticate_uri
    until the old auth_uri option is deprecated from
    the rest of packages, moment in which auth_uri
    references will be removed.

    [0] https://review.openstack.org/#/c/558344/
    [1] https://review.openstack.org/#/c/508522/

    Change-Id: Ie3f59495b1ac43c1a35d912a2da170399652a10e
    Related-Bug: #1761171

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/558831
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=2b662be9a458d71101a1ba4c6a3b45c9cebdd272
Submitter: Zuul
Branch: master

commit 2b662be9a458d71101a1ba4c6a3b45c9cebdd272
Author: Jose Luis Franco Arza <email address hidden>
Date: Wed Apr 4 16:02:01 2018 +0200

    Add www_authenticate_uri option to replace auth_uri.

    auth_uri option has been depreacted in
    favor of www_authenticate_uri from group
    keystone_authtoken in puppet-keystone [0]
    and keystonemiddleware [1].

    This patch adds the new option keeping auth_uri
    references in the templates until the replace
    will be updated in all puppet packages.

    [0] https://review.openstack.org/#/c/558344/
    [1] https://review.openstack.org/#/c/508522/

    Closes-Bug: #1761171
    Change-Id: I804ec73b970844d245dbb0911710ec817359beb0

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/tripleo-heat-templates 9.0.0.0b3 development milestone.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/637018

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/637018
Committed: https://git.openstack.org/cgit/openstack/tripleo-heat-templates/commit/?id=5e629cacd94188330d47113f747aa86c6a39e112
Submitter: Zuul
Branch: master

commit 5e629cacd94188330d47113f747aa86c6a39e112
Author: Goutham Pacha Ravi <email address hidden>
Date: Thu Feb 14 10:00:15 2019 -0800

    Fix keystone opt deprecations in manila manifests

    Add www_authenticate_uri to manila-api manifest.
    Remove auth_uri from api and share manifests because
    puppet-manila no longer supports that option. [1]

    Related-Bug: #1761171
    [1] https://review.openstack.org/#/c/620742/

    Change-Id: I07f6dc5f8e0ae7d0c8060b17b08fd3ae836e073b