When deploying keystone in a HA setup with multiple units and preferred-version=3, the admin_domain_id generated on the leader is not shared with the other units. This causes them to write the default string "admin_domain_id" into policy.json instead of the real value.
This data is generated and stored on disk, and is never transferred to peer units. It seems other information is transferred using peer storage such as in get_admin_password() and get_service_password.
Most likely the admin_domain_id and default_domain_id code should be updated to use peerstorage/leader_(set|get)
lathiat@ubuntu:~/src/charms/xenial/charm-keystone$ juju run --application keystone 'grep \"cloud_admin\" /etc/keystone/policy.json'
- Stdout: |2
"cloud_admin": "rule:admin_required and domain_id:50c240d2979f48f3a73e77b81eb19c60",
UnitId: keystone/0
- Stdout: |2
"cloud_admin": "rule:admin_required and domain_id:admin_domain_id",
UnitId: keystone/1
This is populated by {{ admin_domain_id }}(templates/liberty/policy.json) from KeystoneContext(hooks/keystone_context.py)
hooks/keystone_context.py:KeystoneContext#__call__
ctxt['admin_domain_id'] = (
get_admin_domain_id() or 'admin_domain_id')
hooks/keystone_hooks.py:
def get_admin_domain_id():
return get_file_stored_domain_id(STORED_ADMIN_DOMAIN_ID)
hooks/keystone_utils.py:
def get_file_stored_domain_id(backing_file):
domain_id = None
if os.path.isfile(backing_file):
log("Loading stored domain id from {}".format(backing_file),
level=INFO)
with open(backing_file, 'r') as fd:
domain_id = fd.readline().strip('\n')
return domain_id
Trent, having looked at this I think we need some more info. Specifically, what error does this actually produce on the service side? I presume that whatever error it is it will be only part of the time since requests are load-balanced and those that hit the leader that is configured properly will succeed. If you provide error logs from a service endpoint that is requiring these credentials that would be very useful.