write failure followed by an allocation failure

[Huabin Zheng]

hi all,
   I use memcached 1.4.13 and libmemcached 1.0.4 on CentOS 5.5 64bit. Once the SET operation failed because server temporarily out of memory, the following retry will get write failure error, and it can never auto recovery. My application code looks like this:

    retry = 0;
    do {
      g_rc = memcached_set(g_client, path, path_len, src, file_info.st_size, 18000, flags);
      if (g_rc != MEMCACHED_SUCCESS) {
        printf("Set_Failed: %s, errorcode: %d, retry: %d\n", path, g_rc, retry);
      } else {
        printf("Set_Success: %s\n", path);
        break;
      }
      Wait(10000); // 10000ms
    } while (++retry < 3);

    output:

    Set_Failed: /data/_lcsO_683a0000c7da125d.jpg, errorcode: 17, retry: 0
    Set_Failed: /data_lcsO_683a0000c7da125d.jpg, errorcode: 5, retry: 1
    Set_Failed: /data/_lcsO_683a0000c7da125d.jpg, errorcode: 5, retry: 2
    Set_Failed: /data/PANI_412b00002d21125c.jpg, errorcode: 5, retry: 0
    Set_Failed: /data/PANI_412b00002d21125c.jpg, errorcode: 5, retry: 1

    error code 17:MEMCACHED_MEMORY_ALLOCATION_FAILURE
                    5 :MEMCACHED_WRITE_FAILURE

    so after a failure of allocation, the app is trapped in write failure and can never recovery.

    any hint?

[Huabin Zheng (huabin-zheng)]

some more information. the client is set like this:

  g_client = memcached_create(NULL);
  g_rc = memcached_server_add(g_client, "127.0.0.1", port);
  memcached_behavior_set(g_client, MEMCACHED_BEHAVIOR_DISTRIBUTION,
      MEMCACHED_DISTRIBUTION_CONSISTENT);
  memcached_behavior_set(g_client, MEMCACHED_BEHAVIOR_BINARY_PROTOCOL, 1)

it can easily reproduced by run memcached in gdb, break at one line below "item_alloc" in
    process_bin_update() at memcached.c:2017

set the allocated item, namely it, to 0 to mimic allocation failure, and continue to run.

[Huabin Zheng (huabin-zheng)]

and with this option set:

memcached_behavior_set(g_client, MEMCACHED_BEHAVIOR_RETRY_TIMEOUT, 10);

[Brian Aker (brianaker)]

Thanks, let me look at it.

On Jun 4, 2012, at 8:00, Huabin Zheng <email address hidden> wrote:

> and with this option set:
>
> memcached_behavior_set(g_client, MEMCACHED_BEHAVIOR_RETRY_TIMEOUT, 10);
>
> ** Changed in: libmemcached
> Status: New => Opinion
>
> --
> You received this bug notification because you are subscribed to
> libmemcached.
> https://bugs.launchpad.net/bugs/1007707
>
> Title:
> write failure followed by an allocation failure
>
> Status in libmemcached - A C and C++ client library for memcached:
> Opinion
>
> Bug description:
> hi all,
> I use memcached 1.4.13 and libmemcached 1.0.4 on CentOS 5.5 64bit. Once the SET operation failed because server temporarily out of memory, the following retry will get write failure error, and it can never auto recovery. My application code looks like this:
>
> retry = 0;
> do {
> g_rc = memcached_set(g_client, path, path_len, src, file_info.st_size, 18000, flags);
> if (g_rc != MEMCACHED_SUCCESS) {
> printf("Set_Failed: %s, errorcode: %d, retry: %d\n", path, g_rc, retry);
> } else {
> printf("Set_Success: %s\n", path);
> break;
> }
> Wait(10000); // 10000ms
> } while (++retry < 3);
>
> output:
>
> Set_Failed: /data/_lcsO_683a0000c7da125d.jpg, errorcode: 17, retry: 0
> Set_Failed: /data_lcsO_683a0000c7da125d.jpg, errorcode: 5, retry: 1
> Set_Failed: /data/_lcsO_683a0000c7da125d.jpg, errorcode: 5, retry: 2
> Set_Failed: /data/PANI_412b00002d21125c.jpg, errorcode: 5, retry: 0
> Set_Failed: /data/PANI_412b00002d21125c.jpg, errorcode: 5, retry: 1
>
> error code 17:MEMCACHED_MEMORY_ALLOCATION_FAILURE
> 5 :MEMCACHED_WRITE_FAILURE
>
> so after a failure of allocation, the app is trapped in write
> failure and can never recovery.
>
> any hint?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/libmemcached/+bug/1007707/+subscriptions

[Huabin Zheng (huabin-zheng)]

Hi Brian
Can you reproduce this bug?