Keystone fetches data from Memcache even if caching is explicitly turned off

[Dina Belova]

== Abstract ==

I'm profiling Keystone using OSprofiler tool and the appropriate Keystone+OSprofiler integration changes - https://review.openstack.org/#/q/status:open+project:openstack/keystone+branch:master+topic:osprofiler-support-in-keystone - currently on review. The idea was to analyse how does Keystone use DB/Cache layers.

== Expected vs Observed==

I'm turning off cache via setting

[cache]
enabled = False

I'm expecting all data to be fetched from DB in this case, but I still see gets from Memcache. I mean, *real* gets, not just tries to grab values, but real operations happening with values got from memcache here https://bitbucket.org/zzzeek/dogpile.cache/src/c6913eb143b24b4a886124ff0da5c935ea34e3ac/dogpile/cache/region.py?at=master&fileviewer=file-view-default#region.py-617

Adding OSprofiler HTML report from token issue API call.

[Dina Belova (dbelova)]

Adding keystone log from my env. Really sorry for the mess, there are restarts there, etc due to the experiments

[Morgan Fainberg (mdrnstm)]

I think I know what is going on here.

My Guess is:

* In your environment you ran with keystone connected to memcache

* disabled memcache, but did not clear the memcache server

In this case it is possible keystone was still configured to connect to memcache, so it would read from memcache but new values would not be written to memcache. As thing expired, your memcache server would become more and more empty (or as explicit invalidates occurred). Stale data should not occur in these cases.

The way we disable caching is we use the fn that checks if a value should be cached, and explicitly return False.

The two solutions are:

* set the memcache servers to an empty value when disabling caching globally

* set the dogpile backend to the default null driver

I'll propose a patch that does this in the case that the global cache is set to False; it wont make much of a difference in the case of disabling caching on a specific subsystem until the cache clears/timesout.

[Dina Belova (dbelova)]

Morgan, yes, indeed - I set only

[cache]
enabled = False

without cleaning Memcache servers/ disabling the caching driver. The thing is that I thought in case of turned off caching Keystone should not even try to go to memcache? Why it's even trying to use cache layer if it's globally turned off?

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/304688
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=ea191cacb14818989564ffe1f3727f28be3c3a21
Submitter: Jenkins
Branch: master

commit ea191cacb14818989564ffe1f3727f28be3c3a21
Author: Morgan Fainberg <email address hidden>
Date: Tue Apr 12 08:09:17 2016 -0700

    If caching is globally disabled force dogpile to use the null backend

    Due to the way caching is disabled the SHOULD_CACHE_FN() is used
    to determine if new values are stored in the cache backend, and is
    only called when the regeneration is required. If dogpile is
    configured to connect to a memcache, redis, etc to store data
    it is possible with caching disabled to still pull values from
    the cache (SHOULD_CACHE_FN has no bearing on reads).

    The issue described only impacts the use of the memoization
    decorator.

    This change forces dogpile to use the null backend if caching is
    globally disabled to ensure no data is read from the external
    cache. This will not affect subsystem disabling of cache.

    Even with cache disabled but reads coming from the external cache,
    there stale data is not a concern as invalidates will still be
    processed and the data from the backend will eventually timeout
    in most cases.

    Change-Id: I845b6cc18faa2be516676eeacc574473ca84c995
    Closes-Bug: #1567413

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/oslo.cache 1.7.0 release.