client connection leak to memcached under eventlet due to threadlocal
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
Fix Released
|
Medium
|
Yuriy Taraday | ||
Icehouse |
Won't Fix
|
Medium
|
Unassigned | ||
keystonemiddleware |
Fix Released
|
Medium
|
Morgan Fainberg |
Bug Description
When Keystone configured with memcached as backend and token storage keystone didn't reuse connections to it and starting to fail after having more than 500 connections to the memcached.
Steps to reproduce:
1. Configure keystone with memcached as backend.
2. Create moreless good load (creating of VM's creates a lot of connections) on keystone and watch for connections to memcached using netstat, ex. netstat -an |grep -c ":11211"
Expected behavior:
connections number should be reasonable and be not more that the number of connections to the keystone (ideally :)
Observed bahavior:
Number of connections growing and seems than
1. They didn't reused at all.
2. Lifetime of some connection is 600 seconds.
3. It looks like not all the connections stay for 600 seconds.
<UPDATE from MorganFainberg>
This is specific to deploying under eventlet and the python-memcached library and it's explicit/
Keystone deployed under apache is not affected.
All services deployed with keystonemiddleware that utilize eventlet and memcache for token cache are also affected.
description: | updated |
description: | updated |
tags: | added: memcached |
Changed in keystone: | |
status: | New → Triaged |
importance: | Undecided → Medium |
Changed in keystone: | |
milestone: | none → juno-rc1 |
description: | updated |
Changed in keystone: | |
assignee: | Yuriy Taraday (yorik-sar) → Morgan Fainberg (mdrnstm) |
Changed in keystone: | |
assignee: | Razumovsky Peter (prazumovsky) → Yuriy Taraday (yorik-sar) |
Changed in keystone: | |
assignee: | Yuriy Taraday (yorik-sar) → Morgan Fainberg (mdrnstm) |
Changed in keystone: | |
assignee: | Morgan Fainberg (mdrnstm) → Yuriy Taraday (yorik-sar) |
Changed in keystone: | |
assignee: | Yuriy Taraday (yorik-sar) → Morgan Fainberg (mdrnstm) |
Changed in keystone: | |
assignee: | Morgan Fainberg (mdrnstm) → Yuriy Taraday (yorik-sar) |
Changed in keystone: | |
assignee: | Yuriy Taraday (yorik-sar) → Morgan Fainberg (mdrnstm) |
Changed in keystonemiddleware: | |
milestone: | none → 1.2.0 |
importance: | High → Medium |
Changed in keystonemiddleware: | |
assignee: | Yuriy Taraday (yorik-sar) → Morgan Fainberg (mdrnstm) |
Changed in keystone: | |
assignee: | Morgan Fainberg (mdrnstm) → Yuriy Taraday (yorik-sar) |
Changed in keystonemiddleware: | |
status: | Fix Committed → Fix Released |
Changed in keystone: | |
status: | Fix Committed → Fix Released |
Changed in keystone: | |
milestone: | juno-rc1 → 2014.2 |
Is this using kvs/dogpile or memcached directly? And icehouse or master?