infinite memcached cloning after increasing the pool's version

[Hoonmin Kim]

- libmemcached-0.53, libmemcached-1.0.5

I'm using libmemcached with pooling utility in my multi-threaded server.

A serious problem occurred, when memcached_pool_behavior_set() function was triggered.
The server's performance decreased significantly, and finally I got a site outage for a few minutes.
My site recovered after restarting the server.

After a post mortem I got the reason.

Problem

1. memcached_pool_behavior_set() changes the pool's version.
  - pool->increment_version(); // master->configure.version++
  - copy pool's version to remaining clones

2. In the meanwhile, threads call memcached_pool_release()
  - if (compare_version(released) == false) then memcached_clone()

3. But memcached_clone() does not copy the pool's version.

4. So compare_version() always fails after changing the pool's version.

Fix

add a line somewhere in memcached_clone()
"
new_clone->configure.version= source->configure.version;
"

To test:

1. compie & run the problem below.
2. check the connections with netstat.
3. kill -3 to the running process.
4. re-check the connections with netstat continuously. (see the source port changes)

#include <stdio.h>
#include <signal.h>
#include <stdlib.h>
#include <string.h>
#include <pthread.h>
#include <memcached.h>
#include <pool.h>

#define NUM_THREADS 20

memcached_st *master;
memcached_pool_st *pool;

int running = 1;

void sig_handler(int sig)
{
 switch (sig)
 {
 case SIGINT:
  running = 0;
  break;
 case SIGQUIT:
  fprintf(stderr, "toggled VERIFY_KEY\n");
  memcached_pool_behavior_set(pool, MEMCACHED_BEHAVIOR_VERIFY_KEY, 1);
  break;
 }
}

void *worker_thread(void *ctx)
{
 memcached_pool_st *pool = (memcached_pool_st *)ctx;

 memcached_return_t rc;
 memcached_st *mc;

 while (running)
 {
  mc = memcached_pool_pop(pool, true, &rc);

  if (!mc)
  {
   fprintf(stderr, "failed to fetch a connection from the pool\n");
   sleep(1);
   continue;
  }

  rc = memcached_set(mc, "test:kv", 7, "value", 5, 600, 0);

  rc = memcached_pool_push(pool, mc);

  if (rc != MEMCACHED_SUCCESS)
  {
   fprintf(stderr, "failed to release a connection to the pool\n");
  }
 }
}

int main(int argc, char *argv[])
{
 int i;
 pthread_t pid[NUM_THREADS];

 const char *config_string = "--SERVER=xx.xx.xx.xx:11211 --SERVER=xx.xx.xx.xx:11212 --SERVER=xx.xx.xx.xx:11211 --SERVER=xx.xx.xx.xx:11212";
 master = memcached(config_string, strlen(config_string));

 if (!master) goto RELEASE;

 pool = memcached_pool_create(master, 5, 10);

 if (!pool) goto RELEASE;

 signal(SIGINT, sig_handler);
 signal(SIGQUIT, sig_handler);

 for (i=0; i<NUM_THREADS; i++)
 {
  pthread_create(&pid[i], NULL, worker_thread, (void*)pool);
 }

 for (i=0; i<NUM_THREADS; i++)
 {
  pthread_join(pid[i], NULL);
 }

RELEASE:
 if (pool) memcached_pool_destroy(pool);
 if (master) memcached_free(master);

 return 1;
}