GEARMAN_WORKER_NON_BLOCKING will crash with some values

[Brian Aker]

Hi,

I tried range of values from 2.000 to 10.000.

If I am getting it right $worker->setTimeout($timeout) case when option GEARMAN_WORKER_NON_BLOCKING is set defines interval which $worker->wait() will spend waiting on socket activity.

Segmentation fault will appear if timeout is reached or if job is received. In both cases it's the same fault:

(gdb) run test.php
Starting program: /usr/bin/php test.php
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/mysql.so" does not match "/usr/lib/php5/20090626/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/mysql.so" does not match "/usr/lib/php5/20090626/mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/mysqli.so" does not match "/usr/lib/php5/20090626/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/mysqli.so" does not match "/usr/lib/php5/20090626/mysqli.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug//usr/lib/php5/20090626/pdo_mysql.so" does not match "/usr/lib/php5/20090626/pdo_mysql.so" (CRC mismatch).

warning: the debug information found in "/usr/lib/debug/usr/lib/php5/20090626/pdo_mysql.so" does not match "/usr/lib/php5/20090626/pdo_mysql.so" (CRC mismatch).

[New Thread 0x7fffef826700 (LWP 22165)]
OS: Linux goran 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64
PHP: 5.3.10-1ubuntu3.4
pecl/gearman: 1.1.0
gearman: 1.1.3
[Thread 0x7fffef826700 (LWP 22165) exited]
looping
waiting
wait no more
looping
waiting
wait no more
looping

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff2866328 in gearman_packet_free (packet=0x10a06a8) at libgearman/packet.cc:289
289 if (packet->universal->packet_list == packet)

OS: Linux goran 3.2.0-32-generic #51-Ubuntu SMP Wed Sep 26 21:33:09 UTC 2012 x86_64
PHP: 5.3.10-1ubuntu3.4
pecl/gearman: 1.1.0
gearman: 1.1.3

Let me know if I can provide further information.

[Goran Miskovic (schkovich)]

./configure --enable-assert --disable-libdrizzle --enable-debug

   * CPP Flags: -fvisibility=hidden
   * LIB Flags: -pie
   * Assertions enabled: yes
   * Debug enabled: yes
   * Warnings as failure: no
   * Building with libsqlite3 yes
   * Building with libdrizzle no
   * Building with libmemcached yes
   * Building with libpq no
   * Building with tokyocabinet no
   * Building with libmysql yes
   * make -j: 3
   * VCS checkout: no

[Goran Miskovic (schkovich)]

I been playing a bit. :) Does not mean much to me but hopefully it might save you some time.

Worker used for testing: http://ubuntuone.com/4gcsbr4JB8jLcUJP5GgnYq
Full stack trace having timeout set to 3000: http://ubuntuone.com/324t6c09M61PV0pEnJxys1
Full stack trace no timeout set (seg fault after receiving job): http://ubuntuone.com/0P9CWIvrFbItYkQ2WdHIwM

Stack traces are the same which makes sense. In the first case $worker->wait() will continue after 3 sec e.g. timeout reached while in the second case $worker->wait() will wait on socket activity.

Breakpoint 1, gearman_connection_st::free_private_packet (this=0x10a06d0) at libgearman/connection.cc:338
338 gearman_packet_free(&_packet);
(gdb) print _packet
$17 = {options = {is_allocated = false, complete = false, free_data = false}, magic = GEARMAN_MAGIC_TEXT,
  command = GEARMAN_COMMAND_TEXT, argc = 0 '\000', args_size = 0, data_size = 0, universal = 0x0, next = 0x0, prev = 0x0,
  args = 0x0, data = 0x0, arg = {0x10a083c "exceptions", 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0}, arg_size = {10, 0, 0, 0, 0, 0, 0, 0},
  args_buffer = "\000RES\000\000\000\033\000\000\000\nexceptions", '\000' <repeats 105 times>}
(gdb) step
gearman_packet_free (packet=0x10a0768) at libgearman/packet.cc:271
271 if (packet->args != packet->args_buffer and packet->args)
(gdb) step
280 if (packet->options.free_data and packet->data)
(gdb) step
289 if (packet->universal->packet_list == packet)
(gdb) print packet->universal->packet_list
Cannot access memory at address 0x28
(gdb)

[Brian Aker (brianaker)]

I am a bit confused about the GEARMAN_COMMAND_TEXT/exceptions packet.

[Brian Aker (brianaker)]

I've extended gearman_worker_work_with_TEST() and gearman_worker_work_with_GEARMAN_WORKER_NON_BLOCKING_TEST() to cover the above test case (it should hit trunk sometime soon).

[Ilya Sabelnikov (fruit-dev)]

If I'm right, it is PECL gearman bug https://bugs.php.net/bug.php?id=63041

It sets exception option every time you call ->work();

  PHP_FUNCTION(gearman_worker_work) {
    GEARMAN_ZPMP(RETURN_NULL(), "", &zobj, gearman_worker_ce);

    if (! gearman_worker_set_server_option(&(obj->worker), "exceptions", (sizeof("exceptions") - 1))) {
      GEARMAN_EXCEPTION("Failed to set exception option", 0);
    }

    obj->ret= gearman_worker_work(&(obj->worker));

    // ...
  }

[Ilya Sabelnikov (fruit-dev)]

Here is my patch for gearman PECL extension: http://pastie.org/5440320

[Goran Miskovic (schkovich)]

Hi Ilya, I applied your patch and it's working nicely! Thanks!

[Goran Miskovic (schkovich)]

I filed a bug report against PECL Gearman extension since the bug Ilya is referring to is not referring to the problem I bumped into.

https://bugs.php.net/bug.php?id=63807