cache 'backend' argument description is ambiguous

[Jesse Pretorius]

The oslo.cache backend argument description currently states:

"Dogpile.cache backend module. It is recommended that Memcache or Redis (dogpile.cache.redis) be used in production deployments. For eventlet-based or highly threaded servers, Memcache with pooling (oslo_cache.memcache_pool) is recommended. For low thread servers, dogpile.cache.memcached is recommended. Test environments with a single instance of the server can use the dogpile.cache.memory backend."

So the dogpile.cache.memcached/dogpile.cache.redis backends should be used for production deployments, but the dogpile cache is recommended for low thread servers and the oslo_cache.memcache_pool should be used for high thread servers. I don't understand what the actual recommendation is here.

For a production deployment of a service using uwsgi and a web server, what is the recommendation?
For a production deployment of a service using uwsgi and no web server, what is the recommendation?
For a production deployment of a service using eventlet, what is the recommendation?

Using keystone as an example, the example config file has the same content which does not really help to clarify anything:

https://github.com/openstack/keystone/blob/b7bd6e301964d393ac6835111a08bbf15ba73bc0/etc/keystone.conf.sample#L514-L520

[Bjoern (bjoern-t)]

Adding to this conversation which confuses operators are legacy settings like memcached servers which is derived from the backend arguments https://github.com/openstack/oslo.cache/blob/master/oslo_cache/core.py#L129-L130
I would love to remove this entire code block and just depend on the backend arguments

[Lance Bragstad (lbragstad)]

This change affects keystone but I think it makes sense to only track the fix against the oslo.cache library. There isn't much keystone can do for this fix since the options are generated from oslo.cache and rendered in the sample configuration file.

Improvement to the configuration option help text should positively impact each project using oslo.cache and not just keystone.

Based on the existing help text, it appears that the thread count is the determining factor of what to use. Morgan has some context around oslo.cache and I'm following up with him to see if he had a specific thread count in mind when he wrote that comment [0].

Would it be fair to update the comment with something along the lines of:

- Use oslo_cache.memcache_pool for environments running more than X threads
- Use dogpile.cache.memcached for environments running less than X threads
- Use dogpile.cache.memory for environments isolated to a single process or thread (not recommended for production usage!)

[0] https://github.com/openstack/oslo.cache/blob/67c023c0a7fad02608a4db69bf2c12d00cc8f109/oslo_cache/_opts.py#L32-L45

[Jesse Pretorius (jesse-pretorius)]

Thanks for following up with this Lance. That sounds like a perfectly reasonable aim to go for. Even if it's just a t-shirt size (small, medium, large) - what's important here is knowing which one to use for your dev, test or production environment.

[Lance Bragstad (lbragstad)]

I was able to follow up with Morgan about this in IRC [0]. It sounds like our sizing would be the following

- Use oslo_cache.memcache_pool for environments running 100 threads or more
- Use dogpile.cache.memcached for environments running less than 100 threads but more than a single thread
- Use dogpile.cache.memory for environments isolated to a single process or thread of execution.

This could be accompanied with a statement of tuning like Morgan suggests in IRC. If folks think this is an improvement, I'll get a patch proposed to oslo.cache.

[0] http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-05-17.log.html#t2017-05-17T14:21:19

[Jesse Pretorius (jesse-pretorius)]

Thanks Lance. I guess the only clarification I find we need here would be to provide context to the meaning of the term 'threads' here. It would be helpful to provide context in terms of the configuration of keystone using different configurations (uwsgi, apache/uwsgi, nginx/uwsgi).

It's likely best to provide this in the keystone documentation.

Specifically the confusion here is that there are 'threads' and 'processes'. There may be configurations of many processes, but a single thread per process. I need to have a clear understanding of which recommendation applies in a situation like this, and the keystone documentation should perhaps provide guidance with regards to what the more optimal configurations are for uwsgi, apache/uwsgi, nginx/uwsgi, apache/mod_wsgi, etc.

[Lance Bragstad (lbragstad)]

Jesse, would it be clearer to replace thread with instance of a server running keystone?

[Jesse Pretorius (jesse-pretorius)]

@Lance, well my understanding is that a server has processes and processes have threads. So one 'instance' of keystone (assuming an instance is a 'server') will have one or more processes and each process will have one or more threads. Sticking to the use of the term threads in the context of the server as a whole might be best, while being clear that one server may have many threads.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/465631
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=d6d8c361995f719e7091044abd1f6b123f269cea
Submitter: Jenkins
Branch: master

commit d6d8c361995f719e7091044abd1f6b123f269cea
Author: Lance Bragstad <email address hidden>
Date: Wed May 17 10:24:54 2017 -0500

    Improve backend configuration help text

    This commit adds a little more detail to the backend configuration
    option help text.

    Closes-Bug: 1690756

    Co-Authored-By: ChangBo Guo(gcb) <email address hidden>
    Change-Id: Idf3cc36fe23c6c1d40caff72791eeb41cb1e7173

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/oslo.cache 1.24.0 release.