random crash with pylibmc

[Thorn]

My pylons applications that use pylibmc crash randomly (possibly but unlikely at times of unusually high load). This is a backtrace from one of the crashes (the core file is available if required).

#0 0xf57fe416 in __kernel_vsyscall ()
#1 0xb75fd651 in raise () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7600a82 in abort () from /lib/tls/i686/cmov/libc.so.6
#3 0xb763449d in ?? () from /lib/tls/i686/cmov/libc.so.6
#4 0xb763e591 in ?? () from /lib/tls/i686/cmov/libc.so.6
#5 0xb763fde8 in ?? () from /lib/tls/i686/cmov/libc.so.6
#6 0xb7642ecd in free () from /lib/tls/i686/cmov/libc.so.6
#7 0xb1b2068d in _libmemcached_free (ptr=0x996eea8, mem=0xb0a0f590, context=0x0) at libmemcached/allocators.c:7
#8 0xb1b289ea in libmemcached_free (self=0x0) at libmemcached/common.h:207
#9 memcached_server_free (self=0x0) at libmemcached/server.c:118
#10 0xb1b21e1f in set_last_disconnected_host (ptr=0xb23300b0) at libmemcached/connect.c:479
#11 0xb1b27715 in memcached_quit_server (ptr=0x996f298, io_death=true) at libmemcached/quit.c:82
#12 0xb1b2696b in memcached_io_read (ptr=0x996f298, buffer=0xa43edb8, length=14772, nread=0xb23327bc)
    at libmemcached/io.c:298
#13 0xb1b26b0a in memcached_safe_read (ptr=0x996f298, dta=0xa43edb8, size=48904) at libmemcached/io.c:713
#14 0xb1b27d91 in binary_read_one_response (ptr=0x996f298, buffer=<value optimized out>, buffer_length=350,
    result=0x996ef30) at libmemcached/response.c:405
#15 0xb1b27e89 in memcached_read_one_response (ptr=0x996f298, buffer=0x6 <Address 0x6 out of bounds>, buffer_length=350,
    result=0x4820) at libmemcached/response.c:35
#16 0xb1b23382 in memcached_fetch_result (ptr=0x996eea8, result=0x996ef30, error=0xb2332f84) at libmemcached/fetch.c:64
#17 0xb1b23528 in memcached_fetch (ptr=0x0, key=0x0, key_length=0x0, value_length=0xb2332f8c, flags=0xb2332f88,
    error=0xb2332f84) at libmemcached/fetch.c:16
#18 0xb1b2498e in memcached_get_by_key (ptr=0x996eea8, master_key=0x0, master_key_length=0,
    key=0xa3aa3f4 "coe_org::/ru/page/assist", key_length=24, value_length=0xb2332f8c, flags=0xb2332f88, error=0xb2332f84)
    at libmemcached/get.c:59
#19 0xb1b24c20 in memcached_get (ptr=0x996eea8, key=0xa3aa3f4 "coe_org::/ru/page/assist", key_length=24,
    value_length=0xb2332f8c, flags=0xb2332f88, error=0xb2332f84) at libmemcached/get.c:23
#20 0xb1b36580 in PylibMC_Client_get (self=0x991dc8c, arg=0xa3aa3e0) at _pylibmcmodule.c:376
#21 0x080d3e28 in call_function (f=0xa7545ac, throwflag=0) at Python/ceval.c:4000
#22 PyEval_EvalFrameEx (f=0xa7545ac, throwflag=0) at Python/ceval.c:2665
#23 0x080d63bd in PyEval_EvalCodeEx (co=0x98da968, globals=0x990c13c, locals=0x0, args=0x99a8774, argcount=4,
    kws=0x99a8784, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#24 0x080d4029 in call_function (f=0x99a862c, throwflag=0) at Python/ceval.c:4108
#25 PyEval_EvalFrameEx (f=0x99a862c, throwflag=0) at Python/ceval.c:2665
#26 0x080d63bd in PyEval_EvalCodeEx (co=0x99173c8, globals=0x991213c, locals=0x0, args=0x9cf1118, argcount=1,
    kws=0xa20eb00, kwcount=2, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3252
#27 0x0813445a in function_call (func=0x9918b8c, arg=0x9cf110c, kw=0x9e1f2d4) at Objects/funcobject.c:526
#28 0x0805d9c1 in PyObject_Call (func=0x9918b8c, arg=0x9cf110c, kw=0x9e1f2d4) at Objects/abstract.c:2529
#29 0x080d138d in ext_do_call (f=0x98f6e14, throwflag=0) at Python/ceval.c:4325
etc.

[Thorn (p-thorn-ru)]

libmemcached 0.46, ubuntu 10.04 32-bit, python 2.7.1 (r271:86832) from activestate, pylibmc 1.1.1.

[Brian Aker (brianaker)]

Hi!

On May 5, 2011, at 6:06 AM, Thorn wrote:

> #9 memcached_server_free (self=0x0) at libmemcached/server.c:118
> #10 0xb1b21e1f in set_last_disconnected_host (ptr=0xb23300b0) at libmemcached/connect.c:479

If memcached_server_free() is called on NULL then it will crash. I'm not entirely sure how it would end up in this state, but it is easy enough to make sure this won't happen.

Do you do anything with hosts which are going offline/flapping?

Cheers,
 -Brian

[Thorn (p-thorn-ru)]

Hello,

memcached runs on localhost and has not been stopped or restarted when crashes happened.

[Brian Aker (brianaker)]

set_last_disconnected_host() is trying to disconnect a non-existent host.