Better documentation on what happens when memcached instance fails.

[Denis Shashkov]

All software is self-builded: PHP 5.3.10 + memcached extension 2.1.0, linked with libmemcached 1.0.14.
Running simple PHP-script:
<?php
$m = new Memcached();
$m->setOption(Memcached::OPT_COMPRESSION, false);
$m->setOption(Memcached::OPT_CONNECT_TIMEOUT, 10);
$m->setOption(Memcached::OPT_RETRY_TIMEOUT, 1);
// ketama options
$m->setOption(Memcached::OPT_DISTRIBUTION, Memcached::DISTRIBUTION_CONSISTENT);
$m->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
$m->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 1);
$m->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);

$servers = array();
$servers[] = array('127.0.0.1', 11211, 100);
$servers[] = array('192.168.0.1', 11212, 100);
$m->addServers($servers);

for ( $i=0; $i<100; $i++ ) {
        sleep(1);
        var_dump($m->set('key', 1));
        if ($m->getResultCode() > 0) {
                if ($m->getResultCode() !== Memcached::RES_NOTFOUND ) {
                        echo $m->getResultMessage().PHP_EOL;
                }
        }
}
?>

After several successful cycles, dropping down second server (192.168.0.1).
Expected behavior:
1. After dropping connection and polling timeout, server will be marked as FAULTY.
2. After first connecting attempt, server will be marked as DEAD. Memcached ext./libmemcached will redistribute key to first memcached server (127.0.0.1).

Observed behavior:
1. After dropping connection and polling timeout, server is marked as FAULTY? (It's an assumption, cannot check).
2. Memcached ext./libmemcached open connection to first server (127.0.0.1), SUCCESSFULLY stores the key AND getResultMessage returns "SERVER IS MARKED DEAD". (It's normal)
3. At the next cycle, Memcached ext./libmemcached successfully stores the key. (It's normal, too)
4. At the next and further cycles, Memcached ext./libmemcached return FALSE on set() operation and getResultMessage returns "SERVER IS MARKED DEAD".

It's hard confusing. Why just not using first server further?

I check memcached extension's sources and believe it pass ketama options to libmemcached well.
So I propose that unexpected behavior is a bug. Or maybe I used invalid option set with memcached extension/libmemcached?

[Brian Aker (brianaker)]

Are you using weighting?

On Sun, Mar 24, 2013 at 7:27 PM, Denis Shashkov <email address hidden> wrote:

> ** Description changed:
>
> All software is self-builded: PHP 5.3.10 + memcached extension 2.1.0,
> linked with libmemcached 1.0.14.
> Running simple PHP-script:
> <?php
> $m = new Memcached();
> $m->setOption(Memcached::OPT_COMPRESSION, false);
> $m->setOption(Memcached::OPT_CONNECT_TIMEOUT, 10);
> $m->setOption(Memcached::OPT_RETRY_TIMEOUT, 1);
> // ketama options
> $m->setOption(Memcached::OPT_DISTRIBUTION,
> Memcached::DISTRIBUTION_CONSISTENT);
> $m->setOption(Memcached::OPT_LIBKETAMA_COMPATIBLE, true);
> $m->setOption(Memcached::OPT_SERVER_FAILURE_LIMIT, 1);
> $m->setOption(Memcached::OPT_REMOVE_FAILED_SERVERS, true);
>
> $servers = array();
> $servers[] = array('127.0.0.1', 11211, 100);
> $servers[] = array('192.168.0.1', 11212, 100);
> $m->addServers($servers);
>
> for ( $i=0; $i<100; $i++ ) {
> - sleep(1);
> - var_dump($m->set('key', 1));
> - if ($m->getResultCode() > 0) {
> - if ($m->getResultCode() !== Memcached::RES_NOTFOUND ) {
> - echo $m->getResultMessage().PHP_EOL;
> - }
> - }
> + sleep(1);
> + var_dump($m->set('key', 1));
> + if ($m->getResultCode() > 0) {
> + if ($m->getResultCode() !== Memcached::RES_NOTFOUND ) {
> + echo $m->getResultMessage().PHP_EOL;
> + }
> + }
> }
> ?>
>
> - After several successful cycles, dropping down second server
> (192.168.0.1).
> + After several successful cycles, dropping down second server
> (192.168.0.1).
> Expected behavior:
> 1. After dropping connection and polling timeout, server will be marked
> as FAULTY.
> 2. After first connecting attempt, server will be marked as DEAD.
> Memcached ext./libmemcached will redistribute key to first memcached server
> (127.0.0.1).
>
> Observed behavior:
> 1. After dropping connection and polling timeout, server is marked as
> FAULTY? (It's an assumption, cannot check).
> 2. Memcached ext./libmemcached open connection to first server
> (127.0.0.1), SUCCESSFULLY stores the key AND getResultMessage returns
> "SERVER IS MARKED DEAD". (It's normal)
> 3. At the next cycle, Memcached ext./libmemcached successfully stores
> the key. (It's normal, too)
> - 4. At the next and further cycles, Memcached ext./libmemcached return
> FALSE on set() operation and getResultMessage returns "SERVER IS MARKED
> DEAD".
> + 4. At the next and further cycles, Memcached ext./libmemcached return
> FALSE on set() operation and getResultMessage returns "SERVER IS MARKED
> DEAD".
>
> - It's hard confusing. Why just using first server further?
> + It's hard confusing. Why just not using first server further?
>
> I check memcached extension's sources and believe it pass ketama options
> to libmemcached well.
> - So I propose that unexpected behavior is a bug. Or maybe I use invalid
> option set with memcached extension/libmemcached?
> + So I propose that unexpected behavior is a bug. Or maybe I used invalid
> option set with memcached extension/libmemcached?
...

[Denis Shashkov (bruzh2)]

> Are you using weighting?

Yes, I do.
In the PHP code above I fill the $server array and use equal server weight = 100 (third value in array()).
There is that part:
$servers = array();
$servers[] = array('127.0.0.1', 11211, 100);
$servers[] = array('192.168.0.1', 11212, 100);
$m->addServers($servers);

Also I check second method to initialize server list:
$m->addServer('127.0.0.1', 11211, 100);
$m->addServer('192.168.0.1', 11212, 100);

In memcached extension, first method (addServers) calls memcached_server_list_append_with_weight() in libmemcached. Second one (addServer) calls memcached_server_add_with_weight().

But observed behavior is not changes in general:
1. After dropping server 192.168.0.1 and polling timeout, it's connection (socket #5 in strace2.log) is shutted down. Key is successfully stored to server 127.0.0.1 (socket #4) and getResultMessage returns "SERVER IS MARKED DEAD".
2. At the next cycle, Memcached ext./libmemcached also successfully stores the key to rest server 127.0.0.1
3. At the next and further cycles, Memcached ext./libmemcached return FALSE on set() operation and getResultMessage returns "SERVER IS MARKED DEAD".

I'm not sure that behavior assumes a bug in libmemcached. But I cannot explain for self why memcached extension and libmemcached successfully uses live server two times and not uses it further.

[Denis Shashkov (bruzh2)]

In the PHP code of the first message, I change OPT_RETRY_TIMEOUT to 3 seconds. And got four times of successfully key storing to the rest working server). In the file strace3.log you will see after 'DEAD' message time (13:01:40.742371) from first sendto() operation (13:01:41.742690) to last one (13:01:43.743655) is few less then 3 seconds:

...
13:01:40.742164 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:40.742229 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:40.742295 write(1, "bool(true)\n", 11) = 11
13:01:40.742371 write(1, "SERVER IS MARKED DEAD\n", 22) = 22
...
13:01:41.742690 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:41.742765 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:41.742824 write(1, "bool(true)\n", 11) = 11
...
13:01:42.743163 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:42.743236 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:42.743292 write(1, "bool(true)\n", 11) = 11
...
13:01:43.743655 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:43.743747 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:43.743816 write(1, "bool(true)\n", 11) = 11
...
13:01:44.745050 write(1, "bool(false)\n", 12) = 12
13:01:44.745119 write(1, "SERVER IS MARKED DEAD\n", 22) = 22
...

If I set OPT_RETRY_TIMEOUT to 10 seconds, I got 11 times of successfully key storing after drop down server: one with first DEAD message and 10 times after. And so on. Оbviously, it's because I call sleep() PHP function with parameter in 1 second.

So, I assume that ketama key distributing in libmemcached version 1.0.14 works as follows:
1) after connection to some server is dropping down, this server is marked as DEAD. Keys belonging to this server is 'moved' to the rest servers (I simplify using word 'moved').
2) all operations with this server and IT's KEYS will be dropped after OPT_RETRY_TIMEOUT (in memcached extension) / MEMCACHED_BEHAVIOR_RETRY_TIMEOUT (in libmemcached).

Am I wrong with second thesis?

[Brian Aker (brianaker)]

Yes, your thesis is correct.

[Brian Aker (brianaker)]

BTW if you have any thoughts on how to highlight this in the documentation I am happy to take input.

[Denis Shashkov (bruzh2)]

(I'm sorry, I'm not native english speaker, so I can make stupid mistakes in writing. Hope I don't look impolite.)

I tried to invent some phrases to clarify description of MEMCACHED_BEHAVIOR_RETRY_TIMEOUT, but fails.
I draft some pseudocode understanding how this three behaviour options (MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS,
MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT and MEMCACHED_BEHAVIOR_RETRY_TIMEOUT) implies on libmemcached work.

----------------------
If consistent hashing is not enabled and one server from a server list is down, libmemcached do follows:
 waits for MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT or MEMCACHED_BEHAVIOR_POLL_TIMEOUT and receives 'false' from network operation
 marks server as DISABLED
 // note that connection tries counter = 1 as there was previous successfull connection
 (on this call and next calls with keys mapped to this sever, supposing that server is DISABLED)
 if ( MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS is set ) {
  if ( connection tries counter => MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT ) {
   marks server as DEAD and wipes key hashes. This server will be not used anymore.
   returns false. Corresponding error message is "SERVER IS MARKED DEAD".
  }
 }
 if ( elapsed time after previous operation > MEMCACHED_BEHAVIOR_RETRY_TIMEOUT ) {
  increases connection tries counter
  tries to connect to the server
  if ( server is alive again ) {
   reset DISABLED flag and set connection tries counter to 1
   returns true
  }
 }
 return false. Corresponding error message is "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY".

If consistent hashing is ENABLED and one server from a server list is down, libmemcached do follows:
 waits for MEMCACHED_BEHAVIOR_CONNECT_TIMEOUT or MEMCACHED_BEHAVIOR_POLL_TIMEOUT and receives 'false' from network operation
 marks server as DISABLED
 // note that connection tries counter = 1 as there was previous successfull connection
 (on this call and next calls with keys mapped to this sever, supposing that server is DISABLED)
 if ( MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS is set ) {
  if ( MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT = 1 ) {
   redistributes key hashes to other servers. Connects to them if needed.
   returns true, but with error message is "SERVER IS MARKED DEAD".
  }
  if ( connection tries counter > MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT ) {
   marks server as DEAD and wipes key hashes. This server will be not used anymore.
   If key hashes was redistributed, they will be kept on other servers until expire.
   returns false. Corresponding error message is "SERVER IS MARKED DEAD".
  }
 }
 if ( elapsed time after previous operation > MEMCACHED_BEHAVIOR_RETRY_TIMEOUT ) {
  increases connection tries counter
  tries to connect to the server
  if ( server is alive again ) {
   reset DISABLED flag and set connection tries counter to 1
   returns true
  }
 }
 return false. Corresponding error message is "SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY".

I reach consistent hashing working only in one special case, when it is enabled, MEMCACHED_BEHAVIOR_REMOVE_FAILED_SERVERS is set to true and
MEMCACHED_BEHAVIOR_SERVER_FAILURE_LIMIT is set to 1. And to successfully use redistr...