memcache_socket_timeout is too high

Bug #1731921 reported by Vincent Untz
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
keystonemiddleware
Invalid
Undecided
Vincent Untz
oslo.cache
Fix Released
Medium
Unassigned

Bug Description

The current default for memcache_socket_timeout is 3 seconds. This is very high, and leads to very bad performance when one memcached server is down. See for instance https://bugs.launchpad.net/keystone/+bug/1332058 and https://bugs.launchpad.net/fuel/6.0.x/+bug/1405549 (and probably more).

Moving to memcache_pool does help, but it's not fully helping if you have multiple control nodes and many keystone processes, especially for the first minute or so after the node goes down.

If we decrease memcache_socket_timeout, then things do improve quite a bit. With 1 second, it's okay-ish, but if we move to 0.25 second, then things turn out to be actually nearly normal pretty fast.

I would therefore suggest making this option a float, not an integer, and change the default to either 1 or 0.25 (note that various other socket_timeout settings are using 0.25 out there).

Revision history for this message
Vincent Untz (vuntz) wrote :

I guess it's also valid for memcache_pool_socket_timeout in keystonemiddleware...

Revision history for this message
Vincent Untz (vuntz) wrote :

I submitted https://review.openstack.org/#/c/519353/ to move to a float value, but haven't changed the default there.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to keystonemiddleware (master)

Fix proposed to branch: master
Review: https://review.openstack.org/520745

Changed in keystonemiddleware:
assignee: nobody → Vincent Untz (vuntz)
status: New → In Progress
Revision history for this message
Vincent Untz (vuntz) wrote :

https://review.openstack.org/#/c/520745/ is the keystonemiddleware change.

Revision history for this message
Lance Bragstad (lbragstad) wrote :

Also - keystonemiddleware is in the process of using oslo.cache instead of it's own hand-rolled caching implementation. It might be worth ensuring things work in oslo.cache and have keystonemiddleware consume the fix from oslo.cache instead of having two different fixes.

Changed in oslo.cache:
status: New → Confirmed
Ben Nemec (bnemec)
Changed in oslo.cache:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to oslo.cache (master)

Reviewed: https://review.openstack.org/519353
Committed: https://git.openstack.org/cgit/openstack/oslo.cache/commit/?id=6f320b4b986a73c696e4319ddf9a04f0ac0bec0a
Submitter: Zuul
Branch: master

commit 6f320b4b986a73c696e4319ddf9a04f0ac0bec0a
Author: Vincent Untz <email address hidden>
Date: Mon Nov 13 14:36:19 2017 +0100

    Change memcache_socket_timeout to a float

    It's not unusual to set 0.25 as a socket_timeout value, and this was
    not possible. This can have a dramatic impact for performance when one
    memcached server is down, especially when keystone is involved.

    Change-Id: I01356b7873ad78cb38779f5c5ddd2b932889c911
    Partial-Bug: #1731921

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on keystonemiddleware (master)

Change abandoned by Vincent Untz (<email address hidden>) on branch: master
Review: https://review.openstack.org/520745

Revision history for this message
Lance Bragstad (lbragstad) wrote :

Marking this as invalid for keystonemiddleware since we're switched to using oslo.cache. Thanks for the oslo.cache fix!

Changed in keystonemiddleware:
status: In Progress → Invalid
Vincent Untz (vuntz)
Changed in oslo.cache:
status: Confirmed → Fix Committed
Changed in oslo.cache:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.