Crash when calling exceptions

Bug #1043324 reported by Yonah Russ
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Gearman
Fix Released
High
Brian Aker

Bug Description

# pecl list
Installed packages, channel pecl.php.net:
=========================================
Package Version State
gearman 1.0.3 stable

# php -i | grep -i gearman
gearman
gearman support => enabled
libgearman version => 0.33

core 'core' of 23192: /opt/local/bin/php /opt/local/share/gearmand/gearman_worker.php
----------------- lwp# 1 / thread# 1 --------------------
 fffffd7fff20036a _lwp_kill () + a
 fffffd7fff18a7b1 raise () + 19
 fffffd7fff15fee1 abort () + d9
 fffffd7feff4c6f5 sigsegv_handler () + 125
 fffffd7fff1f73b6 __sighndlr () + 6
 fffffd7fff1ea09c call_user_handler () + 2a4
 fffffd7fff1ea2c3 sigacthandler (6, 0, fffffd7fffdf9c70) + db
 --- called from signal handler with signal 6 (SIGABRT) ---
 fffffd7fff20036a _lwp_kill () + a
 fffffd7fff18a7b1 raise () + 19
 fffffd7fff15fe65 abort () + 5d
 fffffd7fefea54ea ???????? ()
 fffffd7fefea8acf _Z22gearman_request_optionR20gearman_universal_stR16gearman_string_t () + 13f
 fffffd7fefeaad8d gearman_worker_set_server_option () + 1d
 fffffd7fefece3d3 zif_gearman_worker_work () + 63
 000000000074c66a zend_do_fcall_common_helper_SPEC () + a5a
 000000000070e043 execute () + 1c3
 fffffd7feff4114b nr__execute () + 52b
 00000000006ec969 zend_execute_scripts () + 109
 000000000069a41b php_execute_script () + 15b
 000000000077f0d6 main () + 1566
 00000000004c6f7c _start () + 6c
----------------- lwp# 2 / thread# 2 --------------------
 fffffd7fff1ff4da __nanosleep () + a
 fffffd7feff5017a nr__harvest_thread () + 4a
 fffffd7fff1f7024 _thrp_setup () + bc
 fffffd7fff1f72f0 _lwp_start ()

Here is a sanitized version of the code involved:
#!/opt/local/bin/php
<?php
$CONFIG=array();
$pid=getmypid();
echo "Starting\n";

$gmworker= new GearmanWorker();
$gmworker->addServer("gm-local");
$gmworker->addServer("gm-lb");
$gmworker->addFunction("jobFunction","jobFunction");

$gmclient=new GearmanClient();
$gmclient->addServer("gm-lb");
$mysqli=new mysqli('p:db-server', 'user', 'password', 'schema');
if ($mysqli->connect_error) {
  die("MySQL Connection Error: ".$mysqli->connect_error."\n");
}

print "Waiting for job...\n";
while($gmworker->work())
{
  if ($gmworker->returnCode() != GEARMAN_SUCCESS)
  {
    echo "return_code: " . $gmworker->returnCode() . "\n";
    break;
  }
}

function jobFunction($job)
{
 global $gmclient;
 global $pid;
 global $CONFIG;
 global $mysqli;
 $sqlh=$mysqli->prepare('insert into table values(?,?) on duplicate key update field1=values(field1);');
 $workload=json_decode($job->workload(),true);
 $appname=$workload[0];
 $currentJobs=$workload[1];
 $nextJobs=array();
 $object = new ObjectClass();
 echo $pid.": Received job: " . $job->handle() . "\n";
 try{
  $param = array(
   'field' => 'VALUE',
   'field2' => $currentJobs );
  $results = $object->api('/', 'POST', $param);
  if(isset($results) && !empty($results)){
   foreach($results as $result){
    switch($result['code']){
     case 200:
      $resultArr=json_decode($result['body'],true);
      $sqlh->bind_param('is',$resultArr['field'],$resultArr['field1']);
      if($sqlh->execute()){
       print "Upserted ".$sqlh->affected_rows." for ".$resultArr['field']."\n";
      }
      print "Storing Details ".print_r($resultArr,true)." for: ".$resultArr['field']."\n";
      $nextJobs[]=array(
       'field1' =>'value1',
       'field2' => 'value2'
       );
       $jobIndex[]=array($resultArr['field'],'');
      break;
     default:
      echo "Error: ".$result['body']."\n";
      break;
    }
   }
  }
  else{
   error_log("Bad Results: ".print_r($results,true));
  }
 }
 catch(ObjectClassException $e){
  error_log( "ERROR on ".$e);
  echo $pid.": Failed job: ". $job->handle() . "ERROR on ".$e."\n";
  return(-1);
 }
 $job_handle = $gmclient->doBackground("nextJobFunction", json_encode(array($appname,$jobIndex, $nextJobs)));
 if ($gmclient->returnCode() != GEARMAN_SUCCESS){
  error_log("Bad return code");
 }
 else {
  echo "Dispatched Chunk of ".count($nextJobs)." jobs\n";
 }

        echo $pid.": Finished job: " . $job->handle() . "\n";
}

?>

Yonah Russ (c94cs2xkf)
description: updated
Revision history for this message
Herman J. Radtke III (hermanradtke) wrote :

Caused by this code in pecl/gearman:

php_gearman.c:
        if (! gearman_worker_set_server_option(&(obj->worker), "exceptions", (sizeof("exceptions") - 1))) {
                GEARMAN_EXCEPTION("Failed to set exception option", 0);
        }

Revision history for this message
Yonah Russ (c94cs2xkf) wrote :

So is that a bug?
Is there a workaround?
Will upgrading help?
Why do the workers dump core only some of the time?

Revision history for this message
Brian Aker (brianaker) wrote :

Have you enabled exceptions?

Changed in gearmand:
status: New → Incomplete
Revision history for this message
Yonah Russ (c94cs2xkf) wrote :

I guess not.
How do I enable exceptions?
I haven't found anything in the documentation.

Revision history for this message
Brian Aker (brianaker) wrote : Re: [Bug 1043324] Re: core dump using PHP-Gearman 1.0.3 and gearmand 0.33
Download full text (4.9 KiB)

Look in the latest man page on setting exceptions.

Sent from my Ti85

On Sep 3, 2012, at 0:31, Yonah Russ <email address hidden> wrote:

> I guess not.
> How do I enable exceptions?
> I haven't found anything in the documentation.
>
> --
> You received this bug notification because you are subscribed to
> Gearman.
> https://bugs.launchpad.net/bugs/1043324
>
> Title:
> core dump using PHP-Gearman 1.0.3 and gearmand 0.33
>
> Status in Gearman Server and Client Libraries:
> Incomplete
>
> Bug description:
> # pecl list
> Installed packages, channel pecl.php.net:
> =========================================
> Package Version State
> gearman 1.0.3 stable
>
> # php -i | grep -i gearman
> gearman
> gearman support => enabled
> libgearman version => 0.33
>
> core 'core' of 23192: /opt/local/bin/php /opt/local/share/gearmand/gearman_worker.php
> ----------------- lwp# 1 / thread# 1 --------------------
> fffffd7fff20036a _lwp_kill () + a
> fffffd7fff18a7b1 raise () + 19
> fffffd7fff15fee1 abort () + d9
> fffffd7feff4c6f5 sigsegv_handler () + 125
> fffffd7fff1f73b6 __sighndlr () + 6
> fffffd7fff1ea09c call_user_handler () + 2a4
> fffffd7fff1ea2c3 sigacthandler (6, 0, fffffd7fffdf9c70) + db
> --- called from signal handler with signal 6 (SIGABRT) ---
> fffffd7fff20036a _lwp_kill () + a
> fffffd7fff18a7b1 raise () + 19
> fffffd7fff15fe65 abort () + 5d
> fffffd7fefea54ea ???????? ()
> fffffd7fefea8acf _Z22gearman_request_optionR20gearman_universal_stR16gearman_string_t () + 13f
> fffffd7fefeaad8d gearman_worker_set_server_option () + 1d
> fffffd7fefece3d3 zif_gearman_worker_work () + 63
> 000000000074c66a zend_do_fcall_common_helper_SPEC () + a5a
> 000000000070e043 execute () + 1c3
> fffffd7feff4114b nr__execute () + 52b
> 00000000006ec969 zend_execute_scripts () + 109
> 000000000069a41b php_execute_script () + 15b
> 000000000077f0d6 main () + 1566
> 00000000004c6f7c _start () + 6c
> ----------------- lwp# 2 / thread# 2 --------------------
> fffffd7fff1ff4da __nanosleep () + a
> fffffd7feff5017a nr__harvest_thread () + 4a
> fffffd7fff1f7024 _thrp_setup () + bc
> fffffd7fff1f72f0 _lwp_start ()
>
> Here is a sanitized version of the code involved:
> #!/opt/local/bin/php
> <?php
> $CONFIG=array();
> $pid=getmypid();
> echo "Starting\n";
>
> $gmworker= new GearmanWorker();
> $gmworker->addServer("gm-local");
> $gmworker->addServer("gm-lb");
> $gmworker->addFunction("jobFunction","jobFunction");
>
> $gmclient=new GearmanClient();
> $gmclient->addServer("gm-lb");
> $mysqli=new mysqli('p:db-server', 'user', 'password', 'schema');
> if ($mysqli->connect_error) {
> die("MySQL Connection Error: ".$mysqli->connect_error."\n");
> }
>
> print "Waiting for job...\n";
> while($gmworker->work())
> {
> if ($gmworker->returnCode() != GEARMAN_SUCCESS)
> {
> echo "return_code: " . $gmworker->returnCode() . "\n";
> break;
> }
> }
>
> function jobFunction($job)
> {
> global $gmclient;
> global $pid;
> global $CONFIG;
> global $mysqli;
> $sqlh=$mysqli->prepare('insert into table values(?,?) on duplicate key update field1=values(field1);');
> $work...

Read more...

Brian Aker (brianaker)
summary: - core dump using PHP-Gearman 1.0.3 and gearmand 0.33
+ core dump using PHP-Gearman 1.0.3 and gearmand 0.33 calling exceptions
Revision history for this message
Yonah Russ (c94cs2xkf) wrote : Re: core dump using PHP-Gearman 1.0.3 and gearmand 0.33 calling exceptions

I'm still not sure what you mean.
I've combed the PHP-Gearman documentation on http://php.net/manual/en/book.gearman.php?

There are some exception related methods, they are not documented well nor do they have examples:
===========
GearmanClient::setExceptionCallback — Set a callback for worker exceptions

Description
public bool GearmanClient::setExceptionCallback ( callable $callback )
Specifies a function to call when a worker for a task sends an exception.

Parameters
callback - Function to call when the worker throws an exception

Return Values
Returns TRUE on success or FALSE on failure.
===========
All my GearmanClients send only background jobs, nothing waits for a response so I'm not sure that GearmanClient::setExceptionCallback is relevant. I'm honestly not sure how this would work to begin with.
Am I telling the worker to call a function inside the worker on exceptions? and why would I want to do that from the client?
OR
Am I telling the worker to tell me what function to call on exceptions? and why would I need to tell myself what function to call on exceptions? and Wouldn't that only be relevant if I actually wanted a response from the worker?
OR
Am I missing something completely? and Wouldn't an example be helpful?

===========
GearmanJob::sendException — Send exception for running job (exception)

Description
public bool GearmanJob::sendException ( string $exception )
Sends the supplied exception when this job is running.

Parameters
exception -An exception description.

Return Values
Returns TRUE on success or FALSE on failure.
===========

I'm not trying to send any Exceptions for the same reason so I'm not sure GearmanJob::sendException is relevant.

Additionally, I didn't find any mention of /exception/i in relation to GearmanWorker.

If you are by chance near anything more powerful than a Zilog Z80 microprocessor, I'd appreciate more verbose help as to what I'm doing wrong.

Thanks in advance.

Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for Gearman because there has been no activity for 60 days.]

Changed in gearmand:
status: Incomplete → Expired
Revision history for this message
Brian Aker (brianaker) wrote :
Download full text (4.1 KiB)

http://paste.openstack.org/show/35217/

  DEBUG 2013-04-03 17:17:14.889581 [ proc ] 198.101.239.216:60030 packet command GRAB_JOB_UNIQ -> libgearman-server/server.cc:98
  DEBUG 2013-04-03 17:17:14.889633 [ 1 ] Received RUN wakeup event -> libgearman-server/gearmand_thread.cc:607
  DEBUG 2013-04-03 17:17:14.889657 [ 1 ] GEAR length: 0 gearmand_command_t: GEARMAN_COMMAND_NO_JOB -> libgearman-server/plugins/protocol/gear/protocol.cc:274
  DEBUG 2013-04-03 17:17:14.889700 [ 1 ] send() 12 bytes to peer 198.101.239.216:60030 -> libgearman-server/io.cc:275
  DEBUG 2013-04-03 17:17:14.889725 [ 1 ] Sent NO_JOB to 198.101.239.216:29118441 -> libgearman-server/thread.cc:357
  DEBUG 2013-04-03 17:17:14.916524 [ 1 ] 198.101.239.216:60030 Ready POLLIN -> libgearman-server/gearmand_con.cc:118
  DEBUG 2013-04-03 17:17:14.916570 [ 1 ] read 12 bytes -> libgearman-server/io.cc:686
   INFO 2013-04-03 17:17:14.916592 [ 1 ] Gear unpack
  DEBUG 2013-04-03 17:17:14.916632 [ 1 ] GEAR length: 0 gearmand_command_t: GEARMAN_COMMAND_GRAB_JOB_UNIQ -> libgearman-server/plugins/protocol/gear/protocol.cc:248
  DEBUG 2013-04-03 17:17:14.916671 [ 1 ] Received GRAB_JOB_UNIQ 198.101.239.216:29118441 -> libgearman-server/thread.cc:310
  DEBUG 2013-04-03 17:17:14.916707 [ 1 ] 198.101.239.216:60030 Watching POLLIN -> libgearman-server/gearmand_thread.cc:151
  DEBUG 2013-04-03 17:17:14.916742 [ proc ] 198.101.239.216:60030 packet command GRAB_JOB_UNIQ -> libgearman-server/server.cc:98
  DEBUG 2013-04-03 17:17:14.916823 [ 1 ] Received RUN wakeup event -> libgearman-server/gearmand_thread.cc:607
  DEBUG 2013-04-03 17:17:14.916854 [ 1 ] GEAR length: 0 gearmand_command_t: GEARMAN_COMMAND_NO_JOB -> libgearman-server/plugins/protocol/gear/protocol.cc:274
  DEBUG 2013-04-03 17:17:14.916895 [ 1 ] send() 12 bytes to peer 198.101.239.216:60030 -> libgearman-server/io.cc:275
  DEBUG 2013-04-03 17:17:14.916924 [ 1 ] Sent NO_JOB to 198.101.239.216:29118441 -> libgearman-server/thread.cc:357
  DEBUG 2013-04-03 17:17:14.920424 [ 1 ] 198.101.239.216:60030 Ready POLLIN -> libgearman-server/gearmand_con.cc:118
  DEBUG 2013-04-03 17:17:14.920475 [ 1 ] read 12 bytes -> libgearman-server/io.cc:686
   INFO 2013-04-03 17:17:14.920497 [ 1 ] Gear unpack
  DEBUG 2013-04-03 17:17:14.920537 [ 1 ] GEAR length: 0 gearmand_command_t: GEARMAN_COMMAND_PRE_SLEEP -> libgearman-server/plugins/protocol/gear/protocol.cc:248
  DEBUG 2013-04-03 17:17:14.920562 [ 1 ] Received PRE_SLEEP 198.101.239.216:29118441 -> libgearman-server/thread.cc:310
  DEBUG 2013-04-03 17:17:14.920596 [ 1 ] 198.101.239.216:60030 Watching POLLIN -> libgearman-server/gearmand_thread.cc:151
  DEBUG 2013-04-03 17:17:14.920623 [ proc ] 198.101.239.216:60030 packet command PRE_SLEEP -> libgearman-server/server.cc:98
  DEBUG 2013-04-03 17:17:14.920678 [ proc ] delete() gearman_server_packet_st -> libgearman-server/packet.cc:135
  DEBUG 2013-04-03 17:17:14.920829 [ 1 ] 198.101.239.216:60030 Ready POLLIN -> libgearman-server/gearmand_con.cc:118
  DEBUG 2013-04-03 17:17:14.920883 [ 1 ] read 605 bytes -> libgearman-server/io.cc:686
   INFO 2013...

Read more...

Changed in gearmand:
status: Expired → Incomplete
summary: - core dump using PHP-Gearman 1.0.3 and gearmand 0.33 calling exceptions
+ Crash when calling exceptions
Changed in gearmand:
assignee: nobody → Brian Aker (brianaker)
importance: Undecided → High
Revision history for this message
Brian Aker (brianaker) wrote :

Death likely in gearman_server_job_st *server_job= gearman_server_job_get(Server,
                                                                (char *)(packet->arg[0]), (size_t)strlen(packet->arg[0]),
                                                                server_con);

Failure happens before or on:

     gearmand_log_debug(GEARMAN_DEFAULT_LOG_PARAM,
                         "Exception being sent from: %.*s(%lu)",
                         server_job->function->function_name_size, server_job->function->function_name, server_job->function->function_name_size);

Brian Aker (brianaker)
Changed in gearmand:
status: Incomplete → Fix Committed
milestone: none → 1.1.9
Brian Aker (brianaker)
Changed in gearmand:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.