senlin-api requests keystone public endpoint

[Tushar]

host os: centos8 stream
openstack release: victoria (same problem in yoga as well)
openstack deployed using kolla-ansible

when I run openstack cluster list (or any other command of senlin service), it fails with

HttpException: 500: Server Error for url: http://10.27.0.94:8778/v1/clusters?global_project=False, The identity service for defaults: exists but does not have any supported versions.

upon searching in logs, I found that senlin is using keystone public endpoint which is btw using https and is not resolvable in host os (but it should use auth_url endpoint defined in senlin.conf).
------------------------------------- senlin-api.log -------------------------------------
022-08-08 03:05:23.602 30 INFO eventlet.wsgi.server [-] 10.27.0.94,10.27.0.132 - - [08/Aug/2022 03:05:23] "GET / HTTP/1.1" 300 561 0.001869
2022-08-08 03:05:23.612 30 DEBUG senlin.api.middleware.version_negotiation [-] Processing request: GET /v1/clusters Accept: application/json process_request /usr/lib/python3.6/site-packages/senlin/api/middleware/version_negotiation.py:50
2022-08-08 03:05:23.613 30 DEBUG senlin.api.middleware.version_negotiation [-] Matched versioned URI. Version: 1.0 process_request /usr/lib/python3.6/site-packages/senlin/api/middleware/version_negotiation.py:67
2022-08-08 03:05:24.515 30 WARNING keystoneauth.discover [req-44c891e0-6254-4d9d-b0bf-b3fb90ca79c7 0734dae0b2f8414b8efb34d73d491ccc 901e9f8859304cf2b685a9aed8745a39 - - -] Failed to contact the endpoint at https://test-domain.com:5000 for discovery. Fallback to using that endpoint as the base url.
2022-08-08 03:05:24.523 30 WARNING keystoneauth.discover [req-44c891e0-6254-4d9d-b0bf-b3fb90ca79c7 0734dae0b2f8414b8efb34d73d491ccc 901e9f8859304cf2b685a9aed8745a39 - - -] Failed to contact the endpoint at https://test-domain.com:5000 for discovery. Fallback to using that endpoint as the base url.
2022-08-08 03:05:24.523 30 ERROR senlin.drivers.sdk [req-44c891e0-6254-4d9d-b0bf-b3fb90ca79c7 0734dae0b2f8414b8efb34d73d491ccc 901e9f8859304cf2b685a9aed8745a39 - - -] The identity service for defaults: exists but does not have any supported versions.: openstack.exceptions.NotSupported: The identity service for defaults: exists but does not have any supported versions.
2022-08-08 03:05:24.524 30 DEBUG senlin.api.common.serializers [req-44c891e0-6254-4d9d-b0bf-b3fb90ca79c7 0734dae0b2f8414b8efb34d73d491ccc 901e9f8859304cf2b685a9aed8745a39 - - -] JSON response : {"code": 500, "error": {"code": 500, "message": "The identity service for defaults: exists but does not have any supported versions.", "type": "InternalError"}, "explanation": "The server has either erred or is incapable of performing the requested operation.", "title": "Internal Server Error"} to_json /usr/lib/python3.6/site-packages/senlin/api/common/serializers.py:86
2022-08-08 03:05:24.525 30 INFO eventlet.wsgi.server [req-44c891e0-6254-4d9d-b0bf-b3fb90ca79c7 0734dae0b2f8414b8efb34d73d491ccc 901e9f8859304cf2b685a9aed8745a39 - - -] 10.27.0.94,10.27.0.132 - - [08/Aug/2022 03:05:24] "GET /v1/clusters?global_project=False HTTP/1.1" 500 489 0.912918
---------------------------------------------------------------------------------------------
etc/kolla/senlin-api/senlin.conf

[authentication]
auth_url = http://10.27.0.94:35357
service_username = senlin
service_password = PASSWORD
service_project_name = service
service_user_domain = default
service_project_domain = default

[database]
connection = mysql+pymysql://senlin:...@10.27.0.94:3306/senlin
connection_recycle_time = 10
max_pool_size = 1
max_retries = -1

[keystone_authtoken]
www_authenticate_uri = http://10.27.0.94:5000
auth_url = http://10.27.0.94:35357
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = senlin
password = PASSWORD
service_token_roles_required = False
cafile = /etc/pki/tls/certs/ca-bundle.crt
region_name = RegionOne
memcache_security_strategy = ENCRYPT
memcache_secret_key = KEY
memcached_servers = 10.27.0.132:11211
---------------------------------------------------------------

[Bui Doan Dang (dang2705)]

How did you solve your problem I am having similar issue with senlin deployment

[Pham Le Gia Dai (daiplg)]

I tried calling API via `curl` with the internal endpoint and it works, but when I call the public one instead, it behaves like ~tushar1401 description.
does anyone have any ideas?