infinite memcached cloning after increasing the pool's version
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
libmemcached |
Fix Released
|
Medium
|
Brian Aker |
Bug Description
- libmemcached-0.53, libmemcached-1.0.5
I'm using libmemcached with pooling utility in my multi-threaded server.
A serious problem occurred, when memcached_
The server's performance decreased significantly, and finally I got a site outage for a few minutes.
My site recovered after restarting the server.
After a post mortem I got the reason.
Problem
1. memcached_
- pool->increment
- copy pool's version to remaining clones
2. In the meanwhile, threads call memcached_
- if (compare_
3. But memcached_clone() does not copy the pool's version.
4. So compare_version() always fails after changing the pool's version.
Fix
add a line somewhere in memcached_clone()
"
new_clone-
"
To test:
1. compie & run the problem below.
2. check the connections with netstat.
3. kill -3 to the running process.
4. re-check the connections with netstat continuously. (see the source port changes)
#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <memcached.h>
#include <pool.h>
#define NUM_THREADS 20
memcached_st *master;
memcached_pool_st *pool;
int running = 1;
void sig_handler(int sig)
{
switch (sig)
{
case SIGINT:
running = 0;
break;
case SIGQUIT:
fprintf(stderr, "toggled VERIFY_KEY\n");
memcached_
break;
}
}
void *worker_thread(void *ctx)
{
memcached_pool_st *pool = (memcached_pool_st *)ctx;
memcached_return_t rc;
memcached_st *mc;
while (running)
{
mc = memcached_
if (!mc)
{
fprintf(stderr, "failed to fetch a connection from the pool\n");
sleep(1);
continue;
}
rc = memcached_set(mc, "test:kv", 7, "value", 5, 600, 0);
rc = memcached_
if (rc != MEMCACHED_SUCCESS)
{
fprintf(stderr, "failed to release a connection to the pool\n");
}
}
}
int main(int argc, char *argv[])
{
int i;
pthread_t pid[NUM_THREADS];
const char *config_string = "--SERVER=
master = memcached(
if (!master) goto RELEASE;
pool = memcached_
if (!pool) goto RELEASE;
signal(SIGINT, sig_handler);
signal(SIGQUIT, sig_handler);
for (i=0; i<NUM_THREADS; i++)
{
pthread_
}
for (i=0; i<NUM_THREADS; i++)
{
pthread_
}
RELEASE:
if (pool) memcached_
if (master) memcached_
return 1;
}
Changed in libmemcached: | |
assignee: | nobody → Brian Aker (brianaker) |
importance: | Undecided → Medium |
milestone: | none → 1.0.6 |
Changed in libmemcached: | |
status: | New → In Progress |
Changed in libmemcached: | |
status: | In Progress → Fix Committed |
Changed in libmemcached: | |
status: | Fix Committed → Fix Released |