Comment 4 for bug 1086082
Revision history for this message
| Usman Malik (attock) wrote Re: [Bug 1086082] Gearmand dies after starting (loop/out of memory issue?) | #4 |
© 2004
Canonical Ltd.
•
Terms of use
•
Data privacy
•
Contact Launchpad Support
•
Blog
•
Careers
•
System status
•
73416ba
(Get the code!)
It is not clear. The logs are not helpful (in this case when not specifying
--verbose level) and has nothing about the crash.
On Mon, Dec 3, 2012 at 9:25 PM, Brian Aker <email address hidden> wrote:
> Hi,
>
> What took the server down in the first place?
>
> Cheers,
> -Brian
>
> On Dec 3, 2012, at 1:06 PM, Usman Malik <email address hidden> wrote:
>
> > I truncated the table before I could get more stats. But here is the
> > current situation
> >
> > +------
> > | TABLE_NAME | table_rows | data_length | index_length | Size in MB |
> > +------
> > | gearman_queue | 17724 | 205651968 | 2637824 | 198.64 |
> > +------
> >
> > So the average would be ~ 11K
> >
> > So if I had about 200k rows before, that would require ~ 2211 MB of ram.
> >
> >> From the top I pasted above there is:
> >
> > Mem: 1696976k total, 491676k used, 1205300k free, 4644k buffers
> > Swap: 917500k total, 841864k used, 75636k free, 34488k cached
> >
> > If I am doing the math correctly -
> > Ram available: 1205300k + 4644k = 1181 MB
> > Swap available: 75636k + 34488k = 107 MB
> >
> > So it didn't have enough ram space. This doesn't sound like a bug.
> >
> > --
> > You received this bug notification because you are subscribed to
> > Gearman.
> > https:/
> >
> > Title:
> > Gearmand dies after starting (loop/out of memory issue?)
> >
> > Status in Gearman Server and Client Libraries:
> > New
> >
> > Bug description:
> > I had reported a similar bug in the past (which was marked duplicated
> > & had to do with an infinite loop), but I couldn't look it up. I am
> > using mysql with gearmand and after (re)starting gearmand server it
> > does without any error in the log.
> >
> > My guess is that it may have to do with the memory, but there is
> > plenty of memory (& swap) available. This is happening in v0.37,
> > v1.1.1 & v1.1.3.
> >
> > Below I have pasted more information that may help:
> >
> > `top` Result right before the crash:
> > Cpu(s): 1.0%us, 3.3%sy, 0.0%ni, 68.3%id, 27.1%wa, 0.0%hi, 0.0%si,
> 0.3%st
> > Mem: 1696976k total, 491676k used, 1205300k free, 4644k buffers
> > Swap: 917500k total, 841864k used, 75636k free, 34488k cached
> >
> > log:
> > INFO 2012-12-03 18:15:41.241576 [ main ] MySQL queue replay
> > DEBUG 2012-12-03 18:15:41.241903 [ 2 ] Entering thread event loop
> -> libgearman-
> > DEBUG 2012-12-03 18:15:41.242055 [ 3 ] Entering thread event loop
> -> libgearman-
> > DEBUG 2012-12-03 18:15:41.242220 [ 4 ] Entering thread event loop
> -> libgearman-
> > DEBUG 2012-12-03 18:15:41.243491 [ 1 ] Entering thread event loop
> -> libgearman-
> > DEBUG 2012-12-03 18:15:41.243642 [ main ] staring up Epoch thread ->
> libgearman-
> > -- then it adds to queue one by one --
> > DEBUG 2012-12-03 18:16:51.771618 [ main ] Comparing queue 13414 to
> limit 0 for priority 1 -> libgearman-
> > DEBUG 2012-12-03 18:16:51.777002 [ main ] Comparing queue 13415 to
> limit 0 for priority 1 -> libgearman-
> > DEBUG 2012-12-03 18:16:51.803678 [ main ] Comparing queue 13416 to
> limit 0 for priority 1 -> libgearman-
> > -- then it dies --
> >
> > There are about 200K jobs in mysql table.
> >
> > I am running gearmand using init.d script:
> >
> > daemon --pidfile=$pidfile $gearmand -d -l $logfile -j 5 -q MySQL
> > --mysql-user=XXXX --mysql-
> > table=XXXX --verbose DEBUG $OPTIONS
> >
> > (note that DEBUG is only for troubleshooting, otherwise it runs
> > without --verbose)
> >
> > To manage notifications about this bug go to:
> > https:/
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https:/
>
> Title:
> Gearmand dies after starting (loop/out of memory issue?)
>
> Status in Gearman Server and Client Libraries:
> New
>
> Bug description:
> I had reported a similar bug in the past (which was marked duplicated
> & had to do with an infinite loop), but I couldn't look it up. I am
> using mysql with gearmand and after (re)starting gearmand server it
> does without any error in the log.
>
> My guess is that it may have to do with the memory, but there is
> plenty of memory (& swap) available. This is happening in v0.37,
> v1.1.1 & v1.1.3.
>
> Below I have pasted more information that may help:
>
> `top` Result right before the crash:
> Cpu(s): 1.0%us, 3.3%sy, 0.0%ni, 68.3%id, 27.1%wa, 0.0%hi, 0.0%si,
> 0.3%st
> Mem: 1696976k total, 491676k used, 1205300k free, 4644k buffers
> Swap: 917500k total, 841864k used, 75636k free, 34488k cached
>
> log:
> INFO 2012-12-03 18:15:41.241576 [ main ] MySQL queue replay
> DEBUG 2012-12-03 18:15:41.241903 [ 2 ] Entering thread event loop
> -> libgearman-
> DEBUG 2012-12-03 18:15:41.242055 [ 3 ] Entering thread event loop
> -> libgearman-
> DEBUG 2012-12-03 18:15:41.242220 [ 4 ] Entering thread event loop
> -> libgearman-
> DEBUG 2012-12-03 18:15:41.243491 [ 1 ] Entering thread event loop
> -> libgearman-
> DEBUG 2012-12-03 18:15:41.243642 [ main ] staring up Epoch thread ->
> libgearman-
> -- then it adds to queue one by one --
> DEBUG 2012-12-03 18:16:51.771618 [ main ] Comparing queue 13414 to
> limit 0 for priority 1 -> libgearman-
> DEBUG 2012-12-03 18:16:51.777002 [ main ] Comparing queue 13415 to
> limit 0 for priority 1 -> libgearman-
> DEBUG 2012-12-03 18:16:51.803678 [ main ] Comparing queue 13416 to
> limit 0 for priority 1 -> libgearman-
> -- then it dies --
>
> There are about 200K jobs in mysql table.
>
> I am running gearmand using init.d script:
>
> daemon --pidfile=$pidfile $gearmand -d -l $logfile -j 5 -q MySQL
> --mysql-user=XXXX --mysql-
> table=XXXX --verbose DEBUG $OPTIONS
>
> (note that DEBUG is only for troubleshooting, otherwise it runs
> without --verbose)
>
> To manage notifications about this bug go to:
> https:/
>