Comment 8 for bug 1818113

just after identity-credentials-relation-changed hook ran where we were passed authentication credentials from keystone, keystone is immediately not available for the next update-status for the charm to use keystone.

from openstack-service-checks-0.log:
2019-02-28 05:32:06 INFO juju-log identity-credentials:220: Reactive main running for hook identity-credentials-relation-changed
2019-02-28 05:32:12 INFO juju-log identity-credentials:220: render_config: Got credentials for username=nagios
2019-02-28 05:32:28 INFO juju.worker.uniter.operation runhook.go:138 ran "identity-credentials-relation-changed" hook
2019-02-28 05:33:35 INFO juju-log Reactive main running for hook update-status
2019-02-28 05:33:41 ERROR juju-log Hook error:
keystoneauth1.exceptions.http.InternalServerError: An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-058d844c-4b70-42f0-8bda-a7c3b6789bde)

So, somehow keystone charm is able to create the user account from the relation hooks, but the endpoint is not available to authenticate against. This seems either inconsistent or a race condition of clustering of keystone.

I also see a possible race condition where render_config may be called before keystone is ready if nrpe-external-master relation happens first.

Keystone's shared-db isn't even set up until 05:35 or later, so it appears that keystone is able to provide credentials for what "will" be created users when the db is online?

from keystone/0:
2019-02-28 05:35:38 INFO juju-log identity-service:41: database relation's interface, shared-db, is related awaiting the following data from the relationship: database_host, database_password.

Long term, this should probably handled with broker type handling as is done with ceph-mon:client relations. Short term, we'll capture the exception and work around it.