Hi,

Any chance you could attach to the process via gdb and find out what it is dieing on?

Cheers,
	-Brian

On Dec 5, 2012, at 10:06 AM, Usman Malik <email address hidden> wrote:

> It has happened again. I am pretty sure it is due to not having enough
> memory, but the logs have nothing in them even with verbose = ERROR
> 
> Current queue stats:
> 
> Rows: 144875
> Avg Row Length: 10280 bytes
> 
> I think it would be helpful for log something about it.
> 
> Thanks
> 
> On 12/4/12, Usman Malik <email address hidden> wrote:
>> It is not clear. The logs are not helpful (in this case when not specifying
>> --verbose level) and has nothing about the crash.
>> 
>> On Mon, Dec 3, 2012 at 9:25 PM, Brian Aker <email address hidden> wrote:
>> 
>>> Hi,
>>> 
>>> What took the server down in the first place?
>>> 
>>> Cheers,
>>>        -Brian
>>> 
>>> On Dec 3, 2012, at 1:06 PM, Usman Malik <email address hidden> wrote:
>>> 
>>>> I truncated the table before I could get more stats. But here is the
>>>> current situation
>>>> 
>>>> +---------------+------------+-------------+--------------+------------+
>>>> | TABLE_NAME    | table_rows | data_length | index_length | Size in MB
>>>> |
>>>> +---------------+------------+-------------+--------------+------------+
>>>> | gearman_queue |      17724 |   205651968 |      2637824 |     198.64
>>>> |
>>>> +---------------+------------+-------------+--------------+------------+
>>>> 
>>>> So the average would be ~ 11K
>>>> 
>>>> So if I had about 200k rows before, that would require ~ 2211 MB of
>>>> ram.
>>>> 
>>>>> From the top I pasted above there is:
>>>> 
>>>> Mem: 1696976k total, 491676k used, 1205300k free, 4644k buffers
>>>> Swap: 917500k total, 841864k used, 75636k free, 34488k cached
>>>> 
>>>> If I am doing the math correctly -
>>>> Ram available: 1205300k + 4644k = 1181 MB
>>>> Swap available: 75636k +  34488k = 107 MB
>>>> 
>>>> So it didn't have enough ram space. This doesn't sound like a bug.
>>>> 
>>>> --
>>>> You received this bug notification because you are subscribed to
>>>> Gearman.
>>>> https://bugs.launchpad.net/bugs/1086082
>>>> 
>>>> Title:
>>>> Gearmand dies after starting (loop/out of memory issue?)
>>>> 
>>>> Status in Gearman Server and Client Libraries:
>>>> New
>>>> 
>>>> Bug description:
>>>> I had reported a similar bug in the past (which was marked duplicated
>>>> & had to do with an infinite loop), but I couldn't look it up. I am
>>>> using mysql with gearmand and after (re)starting gearmand server it
>>>> does without any error in the log.
>>>> 
>>>> My guess is that it may have to do with the memory, but there is
>>>> plenty of memory (& swap) available. This is happening in v0.37,
>>>> v1.1.1 & v1.1.3.
>>>> 
>>>> Below I have pasted more information that may help:
>>>> 
>>>> `top` Result right before the crash:
>>>> Cpu(s):  1.0%us,  3.3%sy,  0.0%ni, 68.3%id, 27.1%wa,  0.0%hi,  0.0%si,
>>> 0.3%st
>>>> Mem:   1696976k total,   491676k used,  1205300k free,     4644k
>>>> buffers
>>>> Swap:   917500k total,   841864k used,    75636k free,    34488k
>>>> cached
>>>> 
>>>> log:
>>>>   INFO 2012-12-03 18:15:41.241576 [  main ] MySQL queue replay
>>>>   DEBUG 2012-12-03 18:15:41.241903 [     2 ] Entering thread event
>>>> loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>>   DEBUG 2012-12-03 18:15:41.242055 [     3 ] Entering thread event
>>>> loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>>   DEBUG 2012-12-03 18:15:41.242220 [     4 ] Entering thread event
>>>> loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>>   DEBUG 2012-12-03 18:15:41.243491 [     1 ] Entering thread event
>>>> loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>>   DEBUG 2012-12-03 18:15:41.243642 [  main ] staring up Epoch thread
>>>> ->
>>> libgearman-server/timer.cc:61
>>>> -- then it adds to queue one by one --
>>>>   DEBUG 2012-12-03 18:16:51.771618 [  main ] Comparing queue 13414 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>>   DEBUG 2012-12-03 18:16:51.777002 [  main ] Comparing queue 13415 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>>   DEBUG 2012-12-03 18:16:51.803678 [  main ] Comparing queue 13416 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>> -- then it dies --
>>>> 
>>>> There are about 200K jobs in mysql table.
>>>> 
>>>> I am running gearmand using init.d script:
>>>> 
>>>> daemon --pidfile=$pidfile $gearmand -d  -l $logfile -j 5 -q MySQL
>>>> --mysql-user=XXXX --mysql-password=XXXX --mysql-db=XXXX --mysql-
>>>> table=XXXX --verbose DEBUG  $OPTIONS
>>>> 
>>>> (note that DEBUG is only for troubleshooting, otherwise it runs
>>>> without --verbose)
>>>> 
>>>> To manage notifications about this bug go to:
>>>> https://bugs.launchpad.net/gearmand/+bug/1086082/+subscriptions
>>> 
>>> --
>>> You received this bug notification because you are subscribed to the bug
>>> report.
>>> https://bugs.launchpad.net/bugs/1086082
>>> 
>>> Title:
>>>  Gearmand dies after starting (loop/out of memory issue?)
>>> 
>>> Status in Gearman Server and Client Libraries:
>>>  New
>>> 
>>> Bug description:
>>>  I had reported a similar bug in the past (which was marked duplicated
>>>  & had to do with an infinite loop), but I couldn't look it up. I am
>>>  using mysql with gearmand and after (re)starting gearmand server it
>>>  does without any error in the log.
>>> 
>>>  My guess is that it may have to do with the memory, but there is
>>>  plenty of memory (& swap) available. This is happening in v0.37,
>>>  v1.1.1 & v1.1.3.
>>> 
>>>  Below I have pasted more information that may help:
>>> 
>>>  `top` Result right before the crash:
>>>  Cpu(s):  1.0%us,  3.3%sy,  0.0%ni, 68.3%id, 27.1%wa,  0.0%hi,  0.0%si,
>>> 0.3%st
>>>  Mem:   1696976k total,   491676k used,  1205300k free,     4644k
>>> buffers
>>>  Swap:   917500k total,   841864k used,    75636k free,    34488k cached
>>> 
>>>  log:
>>>    INFO 2012-12-03 18:15:41.241576 [  main ] MySQL queue replay
>>>    DEBUG 2012-12-03 18:15:41.241903 [     2 ] Entering thread event loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>    DEBUG 2012-12-03 18:15:41.242055 [     3 ] Entering thread event loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>    DEBUG 2012-12-03 18:15:41.242220 [     4 ] Entering thread event loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>    DEBUG 2012-12-03 18:15:41.243491 [     1 ] Entering thread event loop
>>> -> libgearman-server/gearmand_thread.cc:306
>>>    DEBUG 2012-12-03 18:15:41.243642 [  main ] staring up Epoch thread ->
>>> libgearman-server/timer.cc:61
>>>  -- then it adds to queue one by one --
>>>    DEBUG 2012-12-03 18:16:51.771618 [  main ] Comparing queue 13414 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>    DEBUG 2012-12-03 18:16:51.777002 [  main ] Comparing queue 13415 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>    DEBUG 2012-12-03 18:16:51.803678 [  main ] Comparing queue 13416 to
>>> limit 0 for priority 1 -> libgearman-server/job.c:177
>>>  -- then it dies --
>>> 
>>>  There are about 200K jobs in mysql table.
>>> 
>>>  I am running gearmand using init.d script:
>>> 
>>>  daemon --pidfile=$pidfile $gearmand -d  -l $logfile -j 5 -q MySQL
>>>  --mysql-user=XXXX --mysql-password=XXXX --mysql-db=XXXX --mysql-
>>>  table=XXXX --verbose DEBUG  $OPTIONS
>>> 
>>>  (note that DEBUG is only for troubleshooting, otherwise it runs
>>>  without --verbose)
>>> 
>>> To manage notifications about this bug go to:
>>> https://bugs.launchpad.net/gearmand/+bug/1086082/+subscriptions
>>> 
>> 
> 
> -- 
> You received this bug notification because you are subscribed to
> Gearman.
> https://bugs.launchpad.net/bugs/1086082
> 
> Title:
>  Gearmand dies after starting (loop/out of memory issue?)
> 
> Status in Gearman Server and Client Libraries:
>  New
> 
> Bug description:
>  I had reported a similar bug in the past (which was marked duplicated
>  & had to do with an infinite loop), but I couldn't look it up. I am
>  using mysql with gearmand and after (re)starting gearmand server it
>  does without any error in the log.
> 
>  My guess is that it may have to do with the memory, but there is
>  plenty of memory (& swap) available. This is happening in v0.37,
>  v1.1.1 & v1.1.3.
> 
>  Below I have pasted more information that may help:
> 
>  `top` Result right before the crash:
>  Cpu(s):  1.0%us,  3.3%sy,  0.0%ni, 68.3%id, 27.1%wa,  0.0%hi,  0.0%si,  0.3%st
>  Mem:   1696976k total,   491676k used,  1205300k free,     4644k buffers
>  Swap:   917500k total,   841864k used,    75636k free,    34488k cached
> 
>  log:
>    INFO 2012-12-03 18:15:41.241576 [  main ] MySQL queue replay
>    DEBUG 2012-12-03 18:15:41.241903 [     2 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:306
>    DEBUG 2012-12-03 18:15:41.242055 [     3 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:306
>    DEBUG 2012-12-03 18:15:41.242220 [     4 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:306
>    DEBUG 2012-12-03 18:15:41.243491 [     1 ] Entering thread event loop -> libgearman-server/gearmand_thread.cc:306
>    DEBUG 2012-12-03 18:15:41.243642 [  main ] staring up Epoch thread -> libgearman-server/timer.cc:61
>  -- then it adds to queue one by one --
>    DEBUG 2012-12-03 18:16:51.771618 [  main ] Comparing queue 13414 to limit 0 for priority 1 -> libgearman-server/job.c:177
>    DEBUG 2012-12-03 18:16:51.777002 [  main ] Comparing queue 13415 to limit 0 for priority 1 -> libgearman-server/job.c:177
>    DEBUG 2012-12-03 18:16:51.803678 [  main ] Comparing queue 13416 to limit 0 for priority 1 -> libgearman-server/job.c:177
>  -- then it dies --
> 
>  There are about 200K jobs in mysql table.
> 
>  I am running gearmand using init.d script:
> 
>  daemon --pidfile=$pidfile $gearmand -d  -l $logfile -j 5 -q MySQL
>  --mysql-user=XXXX --mysql-password=XXXX --mysql-db=XXXX --mysql-
>  table=XXXX --verbose DEBUG  $OPTIONS
> 
>  (note that DEBUG is only for troubleshooting, otherwise it runs
>  without --verbose)
> 
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/1086082/+subscriptions