Gearman Server and Client Libraries

GEARMAN_WORKER_NON_BLOCKING will crash with some values

Reported by Brian Aker on 2012-11-13
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Gearman
Undecided
Brian Aker

Bug Description

Hi,

I tried range of values from 2.000 to 10.000.

If I am getting it right $worker->setTimeout($timeout) case when option GEARMAN_WORKER_NON_BLOCKING is set defines interval which $worker->wait() will spend waiting on socket activity.

Segmentation fault will appear if timeout is reached or if job is received. In both cases it's the same fault:

(gdb) run test.php
Starting program: /usr/bin/php test.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/mysql.so" does not match "/usr/lib/php5/20090626/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/mysql.so" does not match "/usr/lib/php5/20090626/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/mysqli.so" does not match "/usr/lib/php5/20090626/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/mysqli.so" does not match "/usr/lib/php5/20090626/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/pdo_mysql.so" does not match "/usr/lib/php5/20090626/pdo_mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/pdo_mysql.so" does not match "/usr/lib/php5/20090626/pdo_mysql.so" (CRC mismatch).

[New Thread 0x7fffef826700 (LWP 22165)]
OS: Linux goran 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64
PHP: 5.3.10-1ubuntu3.4
pecl/gearman: 1.1.0
gearman: 1.1.3
[Thread 0x7fffef826700 (LWP 22165) exited]
looping
waiting
wait no more
looping
waiting
wait no more
looping

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff2866328 in gearman_packet_free (packet=0x10a06a8) at libgearman/packet.cc:289
289 if (packet->universal->packet_list == packet)

OS: Linux goran 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64
PHP: 5.3.10-1ubuntu3.4
pecl/gearman: 1.1.0
gearman: 1.1.3

Let me know if I can provide further information.

schkovich (schkovich) wrote :
schkovich (schkovich) wrote :
schkovich (schkovich) wrote :

./configure --enable-assert --disable-libdrizzle --enable-debug

   * CPP Flags: -fvisibility=hidden
   * LIB Flags: -pie
   * Assertions enabled: yes
   * Debug enabled: yes
   * Warnings as failure: no
   * Building with libsqlite3 yes
   * Building with libdrizzle no
   * Building with libmemcached yes
   * Building with libpq no
   * Building with tokyocabinet no
   * Building with libmysql yes
   * make -j: 3
   * VCS checkout: no

schkovich (schkovich) wrote :

I been playing a bit. :) Does not mean much to me but hopefully it might save you some time.

Worker used for testing: http://ubuntuone.com/4gcsbr4JB8jLcUJP5GgnYq
Full stack trace having timeout set to 3000: http://ubuntuone.com/324t6c09M61PV0pEnJxys1
Full stack trace no timeout set (seg fault after receiving job): http://ubuntuone.com/0P9CWIvrFbItYkQ2WdHIwM

Stack traces are the same which makes sense. In the first case $worker->wait() will continue after 3 sec e.g. timeout reached while in the second case $worker->wait() will wait on socket activity.

Breakpoint 1, gearman_connection_st::free_private_packet (this=0x10a06d0) at libgearman/connection.cc:338
338 gearman_packet_free(&_packet);
(gdb) print _packet
$17 = {options = {is_allocated = false, complete = false, free_data = false}, magic = GEARMAN_MAGIC_TEXT,
  command = GEARMAN_COMMAND_TEXT, argc = 0 '\000', args_size = 0, data_size = 0, universal = 0x0, next = 0x0, prev = 0x0,
  args = 0x0, data = 0x0, arg = {0x10a083c "exceptions", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, arg_size = {10, 0, 0, 0, 0, 0, 0, 0},
  args_buffer = "\000RES\000\000\000\033\000\000\000\nexceptions", '\000' <repeats 105 times>}
(gdb) step
gearman_packet_free (packet=0x10a0768) at libgearman/packet.cc:271
271 if (packet->args != packet->args_buffer and packet->args)
(gdb) step
280 if (packet->options.free_data and packet->data)
(gdb) step
289 if (packet->universal->packet_list == packet)
(gdb) print packet->universal->packet_list
Cannot access memory at address 0x28
(gdb)

Brian Aker (brianaker) on 2012-11-18
Changed in gearmand:
assignee: nobody → Brian Aker (brianaker)
Brian Aker (brianaker) wrote :

I am a bit confused about the GEARMAN_COMMAND_TEXT/exceptions packet.

Brian Aker (brianaker) wrote :

I've extended gearman_worker_work_with_TEST() and gearman_worker_work_with_GEARMAN_WORKER_NON_BLOCKING_TEST() to cover the above test case (it should hit trunk sometime soon).

Brian Aker (brianaker) on 2012-11-21
Changed in gearmand:
status: New → Incomplete
Ilya Sabelnikov (fruit-dev) wrote :

If I'm right, it is PECL gearman bug https://bugs.php.net/bug.php?id=63041

It sets exception option every time you call ->work();

  PHP_FUNCTION(gearman_worker_work) {
    GEARMAN_ZPMP(RETURN_NULL(), "", &zobj, gearman_worker_ce);

    if (! gearman_worker_set_server_option(&(obj->worker), "exceptions", (sizeof("exceptions") - 1))) {
      GEARMAN_EXCEPTION("Failed to set exception option", 0);
    }

    obj->ret= gearman_worker_work(&(obj->worker));

    // ...
  }

Ilya Sabelnikov (fruit-dev) wrote :

Here is my patch for gearman PECL extension: http://pastie.org/5440320

schkovich (schkovich) wrote :

Hi Ilya, I applied your patch and it's working nicely! Thanks!

schkovich (schkovich) wrote :

I filed a bug report against PECL Gearman extension since the bug Ilya is referring to is not referring to the problem I bumped into.

https://bugs.php.net/bug.php?id=63807

Brian Aker (brianaker) on 2013-01-09
Changed in gearmand:
status: Incomplete → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.