Multiple memcached back-end instances breaks caching
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Invalid
|
Undecided
|
Unassigned | ||
oslo.cache |
Fix Released
|
Undecided
|
Morgan Fainberg |
Bug Description
Environment
~~~~~~~~~~~
* Keystone deployed in containers, running on uWSGI (per openstack-ansible defaults)
* Keystone baseline (as provided by OSA 16.0.2): 6a67918f9d5f395
* openstack-ansible version: 16.0.2
* Target OS: Ubuntu 16.04 Xenial
* keystone.conf: http://
Symptom
~~~~~~~
Running keystone against multiple memcached backends (as per OSA standard deployment pattern) results in caching being completely defeated - meaning Keystone's performance is as if caching was disabled.
Switching `backend_argument = url:<cache1ip>
Analysis
~~~~~~~~
Having turned on oslo.cache debugging with the following settings:
`
[DEFAULT]
debug = True
default_log_levels = amqp=WARN,
[cache]
debug_cache_backend = True
`
It becomes obvious that every attempt to retrieve a cached value fails. This appears to be because a different key is generated (through hashing?) despite the paylod (eg: token) being identical.
Evidence
~~~~~~~~
The following log excerpts demonstrate the issue:
* http://
* http://
This is generated through two subsequent attempts to validate the same token (payload shown on the last line of both logs is the same), 3 seconds apart, within the same keystone uWSGI worker process, through the same client invokation:
curl -X GET -H "X-Auth-Token: $ADMIN_TOKEN" -H "X-Subject-Token: $SUBJECT_TOKEN" http://
Yet the cache keys for both requests are different ('5bd08aa07bf8b
Changed in oslo.cache: | |
assignee: | nobody → Morgan Fainberg (mdrnstm) |
I'm able to recreate this with 3 memcached instances, but only with the following configuration:
backend_argument = url:192. 168.122. 243,192. 168.122. 241,192. 168.122. 144:11211
If I rework that configuration option into using memcache_servers I don't see the issue:
memcache_servers = 192.168. 122.243: 11211,192. 168.122. 241:11211, 192.168. 122.144: 11211
I can confirm memcache_servers works as expected by looping a token validation call and seeing traffic get routed to each instance in the ring of memcached servers.
The backend_argument doesn't seem to be working as expected. Looks like you can work around it with memcache_servers for the time being.