Comment 4 for bug 1158676

Revision history for this message
Denis Shashkov (bruzh2) wrote : Re: ketama algorithm doesn't work?

In the PHP code of the first message, I change OPT_RETRY_TIMEOUT to 3 seconds. And got four times of successfully key storing to the rest working server). In the file strace3.log you will see after 'DEAD' message time (13:01:40.742371) from first sendto() operation (13:01:41.742690) to last one (13:01:43.743655) is few less then 3 seconds:

...
13:01:40.742164 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:40.742229 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:40.742295 write(1, "bool(true)\n", 11) = 11
13:01:40.742371 write(1, "SERVER IS MARKED DEAD\n", 22) = 22
...
13:01:41.742690 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:41.742765 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:41.742824 write(1, "bool(true)\n", 11) = 11
...
13:01:42.743163 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:42.743236 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:42.743292 write(1, "bool(true)\n", 11) = 11
...
13:01:43.743655 sendto(4, "set key 1 0 1\r\n1\r\n", 18, MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 18
13:01:43.743747 recvfrom(4, "STORED\r\n", 8196, MSG_DONTWAIT, NULL, NULL) = 8
13:01:43.743816 write(1, "bool(true)\n", 11) = 11
...
13:01:44.745050 write(1, "bool(false)\n", 12) = 12
13:01:44.745119 write(1, "SERVER IS MARKED DEAD\n", 22) = 22
...

If I set OPT_RETRY_TIMEOUT to 10 seconds, I got 11 times of successfully key storing after drop down server: one with first DEAD message and 10 times after. And so on. Оbviously, it's because I call sleep() PHP function with parameter in 1 second.

So, I assume that ketama key distributing in libmemcached version 1.0.14 works as follows:
1) after connection to some server is dropping down, this server is marked as DEAD. Keys belonging to this server is 'moved' to the rest servers (I simplify using word 'moved').
2) all operations with this server and IT's KEYS will be dropped after OPT_RETRY_TIMEOUT (in memcached extension) / MEMCACHED_BEHAVIOR_RETRY_TIMEOUT (in libmemcached).

Am I wrong with second thesis?