memcache_socket_timeout is too high

[Vincent Untz]

The current default for memcache_socket_timeout is 3 seconds. This is very high, and leads to very bad performance when one memcached server is down. See for instance https://bugs.launchpad.net/keystone/+bug/1332058 and https://bugs.launchpad.net/fuel/6.0.x/+bug/1405549 (and probably more).

Moving to memcache_pool does help, but it's not fully helping if you have multiple control nodes and many keystone processes, especially for the first minute or so after the node goes down.

If we decrease memcache_socket_timeout, then things do improve quite a bit. With 1 second, it's okay-ish, but if we move to 0.25 second, then things turn out to be actually nearly normal pretty fast.

I would therefore suggest making this option a float, not an integer, and change the default to either 1 or 0.25 (note that various other socket_timeout settings are using 0.25 out there).

[Vincent Untz (vuntz)]

I guess it's also valid for memcache_pool_socket_timeout in keystonemiddleware...

[Vincent Untz (vuntz)]

I submitted https://review.openstack.org/#/c/519353/ to move to a float value, but haven't changed the default there.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/520745

[Vincent Untz (vuntz)]

https://review.openstack.org/#/c/520745/ is the keystonemiddleware change.

[Lance Bragstad (lbragstad)]

Also - keystonemiddleware is in the process of using oslo.cache instead of it's own hand-rolled caching implementation. It might be worth ensuring things work in oslo.cache and have keystonemiddleware consume the fix from oslo.cache instead of having two different fixes.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/519353
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=6f320b4b986a73c696e4319ddf9a04f0ac0bec0a
Submitter: Zuul
Branch: master

commit 6f320b4b986a73c696e4319ddf9a04f0ac0bec0a
Author: Vincent Untz <email address hidden>
Date: Mon Nov 13 14:36:19 2017 +0100

    Change memcache_socket_timeout to a float

    It's not unusual to set 0.25 as a socket_timeout value, and this was
    not possible. This can have a dramatic impact for performance when one
    memcached server is down, especially when keystone is involved.

    Change-Id: I01356b7873ad78cb38779f5c5ddd2b932889c911
    Partial-Bug: #1731921

[OpenStack Infra (hudson-openstack)]

Change abandoned by Vincent Untz (<email address hidden>) on branch: master
Review: https://review.openstack.org/520745

[Lance Bragstad (lbragstad)]

Marking this as invalid for keystonemiddleware since we're switched to using oslo.cache. Thanks for the oslo.cache fix!