Gearman

  1. Bug #1086082
  2. Comment #5

Comment 5 for bug 1086082

Revision history for this message
Usman Malik (attock) wrote : Re: [Bug 1086082] Gearmand dies after starting (loop/out of memory issue?)

It has happened again. I am pretty sure it is due to not having enough
memory, but the logs have nothing in them even with verbose = ERROR

Current queue stats:

Rows: 144875
Avg Row Length: 10280 bytes

I think it would be helpful for log something about it.

Thanks

On 12/4/12, Usman Malik <email address hidden> wrote:
> It is not clear. The logs are not helpful (in this case when not specifying
> --verbose level) and has nothing about the crash.
>
> On Mon, Dec 3, 2012 at 9:25 PM, Brian Aker <email address hidden> wrote:
>
>> Hi,
>>
>> What took the server down in the first place?
>>
>> Cheers,
>> -Brian
>>
>> On Dec 3, 2012, at 1:06 PM, Usman Malik <email address hidden> wrote:
>>
>> > I truncated the table before I could get more stats. But here is the
>> > current situation
>> >
>> > +---------------+------------+-------------+--------------+------------+
>> > | TABLE_NAME | table_rows | data_length | index_length | Size in MB
>> > |
>> > +---------------+------------+-------------+--------------+------------+
>> > | gearman_queue | 17724 | 205651968 | 2637824 | 198.64
>> > |
>> > +---------------+------------+-------------+--------------+------------+
>> >
>> > So the average would be ~ 11K
>> >
>> > So if I had about 200k rows before, that would require ~ 2211 MB of
>> > ram.
>> >
>> >> From the top I pasted above there is:
>> >
>> > Mem: 1696976k total, 491676k used, 1205300k free, 4644k buffers
>> > Swap: 917500k total, 841864k used, 75636k free, 34488k cached
>> >
>> > If I am doing the math correctly -
>> > Ram available: 1205300k + 4644k = 1181 MB
>> > Swap available: 75636k + 34488k = 107 MB
>> >
>> > So it didn't have enough ram space. This doesn't sound like a bug.
>> >
>> > --
>> > You received this bug notification because you are subscribed to
>> > Gearman.
>> > https://bugs.launchpad.net/bugs/1086082
>> >
>> > Title:
>> > Gearmand dies after starting (loop/out of memory issue?)
>> >
>> > Status in Gearman Server and Client Libraries:
>> > New
>> >
>> > Bug description:
>> > I had reported a similar bug in the past (which was marked duplicated
>> > & had to do with an infinite loop), but I couldn't look it up. I am
>> > using mysql with gearmand and after (re)starting gearmand server it
>> > does without any error in the log.
>> >
>> > My guess is that it may have to do with the memory, but there is
>> > plenty of memory (& swap) available. This is happening in v0.37,
>> > v1.1.1 & v1.1.3.
>> >
>> > Below I have pasted more information that may help:
>> >
>> > `top` Result right before the crash:
>> > Cpu(s): 1.0%us, 3.3%sy, 0.0%ni, 68.3%id, 27.1%wa, 0.0%hi, 0.0%si,
>> 0.3%st
>> > Mem: 1696976k total, 491676k used, 1205300k free, 4644k
>> > buffers
>> > Swap: 917500k total, 841864k used, 75636k free, 34488k
>> > cached
>> >
>> > log:
>> > INFO 2012-12-03 18:15:41.241576 [ main ] MySQL queue replay
>> > DEBUG 2012-12-03 18:15:41.241903 [ 2 ] Entering thread event
>> > loop
>> -> libgearman-server/gearmand_thread.cc:306
>> > DEBUG 2012-12-03 18:15:41.242055 [ 3 ] Entering thread event
>> > loop
>> -> libgearman-server/gearmand_thread.cc:306
>> > DEBUG 2012-12-03 18:15:41.242220 [ 4 ] Entering thread event
>> > loop
>> -> libgearman-server/gearmand_thread.cc:306
>> > DEBUG 2012-12-03 18:15:41.243491 [ 1 ] Entering thread event
>> > loop
>> -> libgearman-server/gearmand_thread.cc:306
>> > DEBUG 2012-12-03 18:15:41.243642 [ main ] staring up Epoch thread
>> > ->
>> libgearman-server/timer.cc:61
>> > -- then it adds to queue one by one --
>> > DEBUG 2012-12-03 18:16:51.771618 [ main ] Comparing queue 13414 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> > DEBUG 2012-12-03 18:16:51.777002 [ main ] Comparing queue 13415 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> > DEBUG 2012-12-03 18:16:51.803678 [ main ] Comparing queue 13416 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> > -- then it dies --
>> >
>> > There are about 200K jobs in mysql table.
>> >
>> > I am running gearmand using init.d script:
>> >
>> > daemon --pidfile=$pidfile $gearmand -d -l $logfile -j 5 -q MySQL
>> > --mysql-user=XXXX --mysql-password=XXXX --mysql-db=XXXX --mysql-
>> > table=XXXX --verbose DEBUG $OPTIONS
>> >
>> > (note that DEBUG is only for troubleshooting, otherwise it runs
>> > without --verbose)
>> >
>> > To manage notifications about this bug go to:
>> > https://bugs.launchpad.net/gearmand/+bug/1086082/+subscriptions
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1086082
>>
>> Title:
>> Gearmand dies after starting (loop/out of memory issue?)
>>
>> Status in Gearman Server and Client Libraries:
>> New
>>
>> Bug description:
>> I had reported a similar bug in the past (which was marked duplicated
>> & had to do with an infinite loop), but I couldn't look it up. I am
>> using mysql with gearmand and after (re)starting gearmand server it
>> does without any error in the log.
>>
>> My guess is that it may have to do with the memory, but there is
>> plenty of memory (& swap) available. This is happening in v0.37,
>> v1.1.1 & v1.1.3.
>>
>> Below I have pasted more information that may help:
>>
>> `top` Result right before the crash:
>> Cpu(s): 1.0%us, 3.3%sy, 0.0%ni, 68.3%id, 27.1%wa, 0.0%hi, 0.0%si,
>> 0.3%st
>> Mem: 1696976k total, 491676k used, 1205300k free, 4644k
>> buffers
>> Swap: 917500k total, 841864k used, 75636k free, 34488k cached
>>
>> log:
>> INFO 2012-12-03 18:15:41.241576 [ main ] MySQL queue replay
>> DEBUG 2012-12-03 18:15:41.241903 [ 2 ] Entering thread event loop
>> -> libgearman-server/gearmand_thread.cc:306
>> DEBUG 2012-12-03 18:15:41.242055 [ 3 ] Entering thread event loop
>> -> libgearman-server/gearmand_thread.cc:306
>> DEBUG 2012-12-03 18:15:41.242220 [ 4 ] Entering thread event loop
>> -> libgearman-server/gearmand_thread.cc:306
>> DEBUG 2012-12-03 18:15:41.243491 [ 1 ] Entering thread event loop
>> -> libgearman-server/gearmand_thread.cc:306
>> DEBUG 2012-12-03 18:15:41.243642 [ main ] staring up Epoch thread ->
>> libgearman-server/timer.cc:61
>> -- then it adds to queue one by one --
>> DEBUG 2012-12-03 18:16:51.771618 [ main ] Comparing queue 13414 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> DEBUG 2012-12-03 18:16:51.777002 [ main ] Comparing queue 13415 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> DEBUG 2012-12-03 18:16:51.803678 [ main ] Comparing queue 13416 to
>> limit 0 for priority 1 -> libgearman-server/job.c:177
>> -- then it dies --
>>
>> There are about 200K jobs in mysql table.
>>
>> I am running gearmand using init.d script:
>>
>> daemon --pidfile=$pidfile $gearmand -d -l $logfile -j 5 -q MySQL
>> --mysql-user=XXXX --mysql-password=XXXX --mysql-db=XXXX --mysql-
>> table=XXXX --verbose DEBUG $OPTIONS
>>
>> (note that DEBUG is only for troubleshooting, otherwise it runs
>> without --verbose)
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/gearmand/+bug/1086082/+subscriptions
>>
>