Ceilometer depends on deprecated admin endpoint of keystone

[Nico Keller]

Error occurs on the Devstack (Release Yoga) using the ceilometer plugin:

--- local.conf ---
...
enable_plugin ceilometer https://opendev.org/openstack/ceilometer stable/yoga
...
---

After investigating the ceilometer notification-agent log, keystoneauth1 does not found the admin entity

---
Jun 27 05:34:27 devstack-01 ceilometer-agent-notification[920606]: 2022-06-27 05:34:27.178 920606 ERROR ceilometer.pipeline.sample keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found
Jun 27 05:34:27 devstack-01 ceilometer-agent-notification[920606]: 2022-06-27 05:34:27.178 920606 ERROR ceilometer.pipeline.sample
---

Looking at the (default) endpoints [openstack endpoint list] there is no keystone identity endpoint with an admin interface.

The following commit gives indications that the keystone admin endpoint has no functional difference anymore from the other endpoints with the v3 API, but heat still depends on it. I assume that there could be a similar problem with ceilometer on devstack:
--> https://review.opendev.org/c/openstack/openstacksdk/+/777343

The latest Keystone documentation also provides information about it:
--> https://docs.openstack.org/keystone/latest/contributor/http-api.html

No configuration changes (e.g. set the interface in the ceilometer.conf [keystone_authtoken] to public) made it possible to set a particular auth-url + interface.

Maybe this Bug is related to the problem. It assumed that the appropriate endpoint selection is not done and therefore the the admin endpoint is selected:
--> https://bugs.launchpad.net/keystoneauth/+bug/1930194

Workaround to get rid of the error is to create the required admin endpoint.

[Yadnesh Kulkarni (ykulkarn)]

which version of keystoneauth is installed ?
pip list | grep keystoneauth

I was able to reproduce it with 4.5.0

could you try upgrading it to 4.6.0 and re-installing the stack.

[Nico Keller (nicokeller)]

Thank you. The version I used was indeed 4.5.0.

I've updated keystoneauth to version 4.6.0.
The Error described above seems to be gone, but instead the adoh-client service seems to throw the same error:

---
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource Traceback (most recent call last):
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 916, in _action_recorder
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource yield
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 1028, in _do_action
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource yield from self.action_handler_task(action, args=handler_args)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 970, in action_handler_task
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource handler_data = handler(*args)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/opt/stack/heat/heat/engine/resources/openstack/aodh/gnocchi/alarm.py", line 111, in handle_create
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource alarm = self.client().alarm.create(props)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/usr/local/lib/python3.8/dist-packages/aodhclient/v2/alarm.py", line 100, in create
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource return self._post(
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/usr/local/lib/python3.8/dist-packages/aodhclient/v2/base.py", line 41, in _post
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource return self.client.api.post(*args, **kwargs)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/home/ubuntu/.local/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 401, in post
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource return self.request(url, 'POST', **kwargs)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource File "/usr/local/lib/python3.8/dist-packages/aodhclient/client.py", line 40, in request
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource raise exceptions.from_response(resp, url, method)
Jul 21 07:55:27 devstack-01 heat-engine[2170585]: ERROR heat.engine.resource aodhclient.exceptions.ClientException: admin endpoint for identity service not found (HTTP 500) (Request-ID: req-7a25bcbb-2a37-43d9-bb28-b9f7141ba9b7)

---

I looked up in the config file (aodh.conf), but the interface configured is public.
I have no clue whether there is a misconfiguration or an inappropriate endpoint selection.

[Christian Rohmann (christian-rohmann)]

Yadnesh did you get a chance to look into this again?