hooks/keystone_utils.py:service_endpoint_dict fails in case v2 Keystone API is used

[Dmitrii Shcherbakov]

https://openstack-ci-reports.ubuntu.com/artifacts/test_charm_pipeline_func_full/openstack/charm-cinder/743343/1/6490/index.html

A xenial-pike deployment (https://review.opendev.org/#/c/743343/) experienced an issue with a Keystone unit not being able to retrieve the 'url' parameter from the endpoint object dict.

The newly added code that attempts to do that

https://opendev.org/openstack/charm-keystone/src/commit/819c41acdac0e54331c00832ce8a90537fcd57b9/hooks/keystone_utils.py#L2109-L2111

is called conditionally

https://opendev.org/openstack/charm-keystone/src/commit/819c41acdac0e54331c00832ce8a90537fcd57b9/hooks/keystone_utils.py#L2151-L2154

Therefore, the fact that there was one successful full recheck (https://review.opendev.org/#/c/741617/) does not protect against an intermittent issue.

Looking at the failure more closely,

unit-keystone-0.log

2020-07-28 04:31:36 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 2152, in send_id_service_notifications
2020-07-28 04:31:36 DEBUG identity-service-relation-changed service_endpoint = service_endpoint_dict(service_name)
2020-07-28 04:31:36 DEBUG identity-service-relation-changed File "/var/lib/juju/agents/unit-keystone-0/charm/hooks/keystone_utils.py", line 2111, in service_endpoint_dict
2020-07-28 04:31:36 DEBUG identity-service-relation-changed endpoint_dict[endpoint['interface']] = endpoint['url']
2020-07-28 04:31:36 DEBUG identity-service-relation-changed KeyError: 'url'
2020-07-28 04:31:36 ERROR juju.worker.uniter.operation runhook.go:132 hook "identity-service-relation-changed" failed: exit status 1

keystone.log has warnings about the usage of the V2 API:

(oslo_log.versionutils): 2020-07-28 04:30:09,922 WARNING Deprecated: get_endpoints of the v2 API is deprecated as of Mitaka in favor of a similar function in the v3 API and may be removed in Queens.

For xenial-pike, unless a preferred-api-version config is set, v2 API is used (test bundles do not override it):

https://opendev.org/openstack/charm-keystone/src/commit/819c41acdac0e54331c00832ce8a90537fcd57b9/hooks/keystone_utils.py#L1468-L1479

v2 endpoint objects do not seem to contain a 'url' parameter:

https://opendev.org/openstack/keystone/src/branch/stable/pike/api-ref/source/v2-admin/samples/admin/endpoint-list-response.json

https://opendev.org/openstack/keystone/src/branch/stable/pike/keystone/catalog/controllers.py#L72
https://opendev.org/openstack/keystone/src/branch/stable/pike/keystone/catalog/controllers.py#L102-L103

So we need to fix the endpoint querying code for v2 usage cases.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.opendev.org/743488

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.opendev.org/743488
Committed: https://git.openstack.org/cgit/openstack/charm-keystone/commit/?id=e02f0590a70ccc0c7e8f31d6db6116afa2ca543f
Submitter: Zuul
Branch: master

commit e02f0590a70ccc0c7e8f31d6db6116afa2ca543f
Author: James Page <email address hidden>
Date: Tue Jul 28 10:13:55 2020 +0100

    endpoint notification v2 API compatibility

    Fallback to v2 API behaviour for processing endpoints for older
    OpenStack releases where the v2 API is still the default.

    Change-Id: Ieb9afed0a6442fac48f8e1ccc0f5c34626a3be18
    Closes-Bug: 1889180