Creating a host fails with 404 from keystone

[Sam Morrison]

Get the following error when doing a blazar host-create

2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server [-] Exception during message handling: keystoneauth1.exceptions.http.NotFound: Not Found (HTTP 404) (Request-ID: req-c34c6a1b-d47a-4fd6-8755-90a852cedd1a)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/blazar/blazar/utils/service.py", line 77, in run_method
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return method(**kwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/blazar/blazar/plugins/oshosts/host_plugin.py", line 366, in create_computehost
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server host_details['hypervisor_hostname'])
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/blazar/blazar/utils/openstack/placement.py", line 198, in create_reservation_provider
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server host_rp = self.get_resource_provider(host_name)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/blazar/blazar/utils/openstack/placement.py", line 106, in get_resource_provider
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server resp = self.get(url)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/blazar/blazar/utils/openstack/placement.py", line 83, in get
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server microversion=microversion)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 375, in get
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 237, in request
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/session.py", line 704, in request
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/session.py", line 1097, in get_auth_headers
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server token = self.get_token(session)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 178, in get_auth_ref
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server authenticated=False, log=False, **rkwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/session.py", line 1045, in post
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server File "/opt/stein/lib/python3.6/site-packages/keystoneauth1/session.py", line 890, in request
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2019-12-17 17:20:27.677 32512 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.NotFound: Not Found (HTTP 404) (Request-ID: req-c34c6a1b-d47a-4fd6-8755-90a852cedd1a)

I tracked this down to the URL it is generating for keystone which ends up being:

http://keystone.example.com:5000//v3/auth/token

The double slash causes the 404. Looks like need to handle not setting an auth prefix better. I think setting the keystone auth_url in the config would make this better as opposed to setting each part of the url individually.

[Pierre Riteau (priteau)]

Hi Sam,

Thanks a lot for the report. I assume that [DEFAULT]/os_auth_prefix is unset in your environment and uses the default value of empty string? I am surprised that the double-slash causes an issue though, as I have not seen this issue on any Blazar deployment to date. Example of a blazar.conf configuration that works:

os_auth_host = 10.225.0.1
os_auth_port = 35357
os_auth_protocol = http
os_auth_version = v3

Still, I agree that this is unnecessarily complex. We will propose a change to simplify the configuration of the Keystone URL.

[Tetsuro Nakamura (tetsuro0907)]

@Sam

Does https://review.opendev.org/#/c/719456 help you?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/753316

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/719456
Committed: https://git.openstack.org/cgit/openstack/blazar/commit/?id=3167277843986efe50a03dc15e04acf2ef742fef
Submitter: Zuul
Branch: master

commit 3167277843986efe50a03dc15e04acf2ef742fef
Author: Pierre Riteau <email address hidden>
Date: Mon Apr 13 15:04:58 2020 +0200

    Avoid double slash in Keystone auth_url

    The auth_url parameter is constructed from several configuration values,
    including os_auth_prefix which defaults to an empty string. This can
    result in a Keystone auth_url containing a double slash, which appears
    to cause issues in some deployments.

    Change-Id: I9e78e5ef1c29c88d3dc16447a78a6e53c5a71567
    Closes-Bug: #1856658

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/753316
Committed: https://git.openstack.org/cgit/openstack/blazar/commit/?id=9c5a5830a7be50e0114fef84b4596ba72f006bd6
Submitter: Zuul
Branch: stable/ussuri

commit 9c5a5830a7be50e0114fef84b4596ba72f006bd6
Author: Pierre Riteau <email address hidden>
Date: Mon Apr 13 15:04:58 2020 +0200

    Avoid double slash in Keystone auth_url

    The auth_url parameter is constructed from several configuration values,
    including os_auth_prefix which defaults to an empty string. This can
    result in a Keystone auth_url containing a double slash, which appears
    to cause issues in some deployments.

    Change-Id: I9e78e5ef1c29c88d3dc16447a78a6e53c5a71567
    Closes-Bug: #1856658
    (cherry picked from commit 3167277843986efe50a03dc15e04acf2ef742fef)

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/blazar 5.0.1 release.