This is a very difficult to reproduce bug but occurs nevertheless and can be observed when we switch the backend drivers.
Steps to reproduce:
1. Switch the backened driver in keystone conf file from one driver to another . Restart keystone
2. Immediately (if you wait for few seconds, this cannot be reproduced) , make calls that in turn access the keystone methods /keystone/identity/core >> get_group and list_users_in_group. It doesn't have to be exactly these two. It can be any two similar methods in identity.core.py which uses the @domains_configured decorator.
https://github.com/openstack/keystone/blob/master/keystone/identity/core.py#L100
Invoke two methods that use this decorators and these method invocations must be almost parallel. Both the methods hit the following flow where the race condition occurs:
def domains_configured(f):
"""Wraps API calls to lazy load domain configs after init.
"""
@functools.wraps(f)
def wrapper(self, *args, **kwargs):
if (not self.domain_configs.configured and
CONF.identity.domain_specific_drivers_enabled):
self.domain_configs.setup_domain_drivers(
self.driver, self.resource_api)
else:
LOG.error('domains will not be configured')
return f(self, *args, **kwargs)
return wrapper
def setup_domain_drivers(self, standard_driver, resource_api):
# This is called by the api call wrapper
self.configured = True
self.driver = standard_driver
.....
When the first call is placed, it sets self.configured to True and then proceeds towards loading the driver that corresponds to the domain-..However, the second request call assumes the the driver load is already complete (purely based on the value set to self.configured - which is True even though driver is not really loaded). It thus ends up using the default driver (ie driver which is not domain specific ) and retrieves the values.
There should be some synchronization logic added inside domains_configured (or one of the internal methods) so that incorrect backend driver is not used by a request.
I presume you are running Keystone under Apache, configured with multi-thread and multi-process? This problem may not be reproducible under devstack with single-thread, multi-process configuration.
WSGIDaemonProcess keystone-admin processes=5 threads=1 user=gyee display-