Assertion in libgearman/client.cc:1699 in latest trunk

[Taneli Leppä]

I'm getting this using php-gearman module (both 0.7.0 and 0.8.0) compiled against the latest libgearman trunk and gearmand:

libgearman/client.cc:1669: void* _client_do(gearman_client_st*, gearman_command_t, const char*, size_t, gearman_unique_t&, const void*, size_t, size_t*, gearman_return_t*): Assertion `client->task_list' failed.

Gearman 0.20 works fine though. Has there been an API change or is this a bug?

[Brian Aker (brianaker)]

I would check back with trunk and see if this is still occurring.

What was the sequence of calls that created the assert?

[Taneli Leppä (rosmiitto)]

I'll check the latest trunk and I'll try to some sort of call trace for you, if it still fails.

[Brian Aker (brianaker)]

Thanks!

In particular I am wondering if it might be from a combination of do() and add_task() requests.

Cheers,
 -Brian

On May 27, 2011, at 11:03 AM, smo wrote:

> I'll check the latest trunk and I'll try to some sort of call trace for
> you, if it still fails.
>
> --
> You received this bug notification because you are subscribed to
> Gearman.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Incomplete
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?

[Taneli Leppä (rosmiitto)]

Allright, I built the latest trunk as of today. Here's the test script:

  $gc = new \GearmanClient();
  $gc->addServer('127.0.0.1');
  $gc->do('removed', '{"version":"1.1","method":"samplemethod","params":[],"id":"M4oGoeVfSFWd7s2AIrX4KA"}');

Result is:

php: libgearman/client.cc:1669: void* _client_do(gearman_client_st*, gearman_command_t, const char*, size_t, gearman_unique_t&, const void*, size_t, size_t*, gearman_return_t*): Assertion `client->task_list' failed.

The backtrace is:

#0 0x00007ffff5bf9945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00007ffff5bfb125 in abort () at abort.c:92
#2 0x00007ffff5bf2955 in __assert_fail (assertion=0x7fffed256595 "client->task_list", file=<value optimized out>, line=1669,
    function=<value optimized out>) at assert.c:81
#3 0x00007fffed24f63c in _client_do (client=0x14ebb88, command=<value optimized out>, function_name=<value optimized out>,
    function_name_length=<value optimized out>, unique=<value optimized out>, workload=<value optimized out>, workload_size=89,
    result_size=0x7fffffffba18, ret_ptr=0x14ebb80) at libgearman/client.cc:1669
#4 0x00007fffed24f85b in gearman_client_do (client=0x14ebb88, function_name=0x14ed320 "removed", unique=0x0, workload=0x14ec1d8,
    workload_size=89, result_size=0x7fffffffba18, ret_ptr=0x14ebb80) at libgearman/client.cc:357
#5 0x00007fffed46a29e in zif_gearman_client_do (ht=<value optimized out>, return_value=0x14ec248, return_value_ptr=<value optimized out>,
    this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /builddir/build/BUILD/gearman-0.8.0/php_gearman.c:1937
#6 0x000000000081e788 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:316
#7 0x00000000007f5c70 in execute (op_array=0x14eb0b0) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:107
#8 0x00000000007d014f in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-5.3.6/Zend/zend.c:1194
#9 0x000000000077d5f8 in php_execute_script (primary_file=0x7fffffffe310) at /usr/src/debug/php-5.3.6/main/main.c:2268
#10 0x000000000085a22e in main (argc=2, argv=0x7fffffffe5a8) at /usr/src/debug/php-5.3.6/sapi/cli/php_cli.c:1193

The server logs don't show anything beyond a connection and then EOF from the client.

[Brian Aker (brianaker)]

Thanks, let me take a look at it.

Is the client setup to be non-blocking?

On Jun 6, 2011, at 5:40 AM, smo wrote:

> Allright, I built the latest trunk as of today. Here's the test script:
>
> $gc = new \GearmanClient();
> $gc->addServer('127.0.0.1');
> $gc->do('removed', '{"version":"1.1","method":"samplemethod","params":[],"id":"M4oGoeVfSFWd7s2AIrX4KA"}');
>
> Result is:
>
> php: libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const void*,
> size_t, size_t*, gearman_return_t*): Assertion `client->task_list'
> failed.
>
> The backtrace is:
>
> #0 0x00007ffff5bf9945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
> #1 0x00007ffff5bfb125 in abort () at abort.c:92
> #2 0x00007ffff5bf2955 in __assert_fail (assertion=0x7fffed256595 "client->task_list", file=<value optimized out>, line=1669,
> function=<value optimized out>) at assert.c:81
> #3 0x00007fffed24f63c in _client_do (client=0x14ebb88, command=<value optimized out>, function_name=<value optimized out>,
> function_name_length=<value optimized out>, unique=<value optimized out>, workload=<value optimized out>, workload_size=89,
> result_size=0x7fffffffba18, ret_ptr=0x14ebb80) at libgearman/client.cc:1669
> #4 0x00007fffed24f85b in gearman_client_do (client=0x14ebb88, function_name=0x14ed320 "removed", unique=0x0, workload=0x14ec1d8,
> workload_size=89, result_size=0x7fffffffba18, ret_ptr=0x14ebb80) at libgearman/client.cc:357
> #5 0x00007fffed46a29e in zif_gearman_client_do (ht=<value optimized out>, return_value=0x14ec248, return_value_ptr=<value optimized out>,
> this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /builddir/build/BUILD/gearman-0.8.0/php_gearman.c:1937
> #6 0x000000000081e788 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:316
> #7 0x00000000007f5c70 in execute (op_array=0x14eb0b0) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:107
> #8 0x00000000007d014f in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-5.3.6/Zend/zend.c:1194
> #9 0x000000000077d5f8 in php_execute_script (primary_file=0x7fffffffe310) at /usr/src/debug/php-5.3.6/main/main.c:2268
> #10 0x000000000085a22e in main (argc=2, argv=0x7fffffffe5a8) at /usr/src/debug/php-5.3.6/sapi/cli/php_cli.c:1193
>
> The server logs don't show anything beyond a connection and then EOF
> from the client.
>
> --
> You received this bug notification because you are subscribed to
> Gearman.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Incomplete
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine t...

[Taneli Leppä (rosmiitto)]

I'm pretty sure it's not (it's not the default configuration).

[Taneli Leppä (rosmiitto)]

Yay! Also I was wondering, have you fixed persistent queue when using sqlite yet? The SQL statements for creating the table are a bit in wrong order (key before all the fields etc)? I have a patch right here if it's still so.

[Brian Aker (brianaker)]

Can you try out lp:gearmand and tell me if it resolves the issues you found? I believe everything is fine. I went and added a few additional tests to it, but I am sure that it could use more.

Thanks,
 -Brian

On Jun 8, 2011, at 4:06 PM, smo wrote:

> Yay! Also I was wondering, have you fixed persistent queue when using
> sqlite yet? The SQL statements for creating the table are a bit in wrong
> order (key before all the fields etc)? I have a patch right here if it's
> still so.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Committed
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions

[Taneli Leppä (rosmiitto)]

I tried both branches (lp:gearmand and lp:gearmand/build, not sure what the difference is), but the assertion remains:

php: libgearman/client.cc:1201: void* _client_do(gearman_client_st*, gearman_command_t, const char*, const char*, const void*, size_t, size_t*, gearman_return_t*): Assertion `client->task_list' failed.

[Brian Aker (brianaker)]

Can you pull what is in lp:gearmand and rerun? I am thinking that you might have gotten a stale copy.

On Jun 9, 2011, at 8:03 AM, smo wrote:

> I tried both branches (lp:gearmand and lp:gearmand/build, not sure what
> the difference is), but the assertion remains:
>
> php: libgearman/client.cc:1201: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, const char*, const void*, size_t,
> size_t*, gearman_return_t*): Assertion `client->task_list' failed.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Committed
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions

[Taneli Leppä (rosmiitto)]

Alright, I pulled in the latest lp:gearmand and the result is still:

php: libgearman/client.cc:1251: void* _client_do(gearman_client_st*, gearman_command_t, const char*, const char*, const void*, size_t, size_t*, gearman_return_t*): Assertion `client->task_list' failed.

Backtrace:

(gdb) bt
#0 0x00000039c8832945 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1 0x00000039c8834125 in abort () at abort.c:92
#2 0x00000039c882b955 in __assert_fail (assertion=0x7ffff1a668ca "client->task_list", file=<value optimized out>, line=1251, function=<value optimized out>) at assert.c:81
#3 0x00007ffff1a5d1cc in ?? () from /usr/lib64/libgearman.so.6
#4 0x00007ffff1a5d258 in gearman_client_do () from /usr/lib64/libgearman.so.6
#5 0x00007ffff1c7b29e in zif_gearman_client_do (ht=<value optimized out>, return_value=0x14f0ff0, return_value_ptr=<value optimized out>, this_ptr=<value optimized out>,
    return_value_used=<value optimized out>) at /builddir/build/BUILD/gearman-0.8.0/php_gearman.c:1937
#6 0x000000000081e788 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:316
#7 0x00000000007f5c70 in execute (op_array=0x14efe50) at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:107
#8 0x00000000007d014f in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /usr/src/debug/php-5.3.6/Zend/zend.c:1194
#9 0x000000000077d5f8 in php_execute_script (primary_file=0x7fffffffe310) at /usr/src/debug/php-5.3.6/main/main.c:2268
#10 0x000000000085a22e in main (argc=2, argv=0x7fffffffe5a8) at /usr/src/debug/php-5.3.6/sapi/cli/php_cli.c:1193

[Taneli Leppä (rosmiitto)]

Just built the latest 0.22 trunk a few minutes ago and I'm still getting the error:

php: libgearman/client.cc:1243: void* _client_do(gearman_client_st*, gearman_command_t, const char*, const char*, const void*, size_t, size_t*, gearman_return_t*): Assertion `client->task_list' failed.

There must be something that the PHP library does wrong.

[Taneli Leppä (rosmiitto)]

The PHP client seems to use the option GEARMAN_CLIENT_FREE_TASKS by default, maybe this is the culprit?

[Taneli Leppä (rosmiitto)]

Disabling the assertions causes this:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff1a6129c in gearman_packet_free (packet=0x7fffffffb748) at libgearman/packet.cc:265
265 if (not (packet->universal->options.dont_track_packets))
Missing separate debuginfos, use: debuginfo-install v8-3.2.10.3-1.foo6.x86_64
(gdb) bt
#0 0x00007ffff1a6129c in gearman_packet_free (packet=0x7fffffffb748) at libgearman/packet.cc:265
#1 0x00007ffff1a63519 in gearman_task_free (task=0x7fffffffb700) at libgearman/task.cc:120
#2 0x00007ffff1a5d143 in _client_do (client=0x14f0a58, command=<value optimized out>, function_name=<value optimized out>,
    unique=<value optimized out>, workload_str=<value optimized out>, workload_size=89, result_size=0x7fffffffba08,
    ret_ptr=<value optimized out>) at libgearman/client.cc:1246
#3 0x00007ffff1a5d2c8 in gearman_client_do (client=<value optimized out>, function=<value optimized out>, unique=<value optimized out>,
    workload=<value optimized out>, workload_size=<value optimized out>, result_size=<value optimized out>, ret_ptr=0x14f0a50)
    at libgearman/client.cc:363
#4 0x00007ffff1c7b29e in zif_gearman_client_do (ht=<value optimized out>, return_value=0x14f1120, return_value_ptr=<value optimized out>,
    this_ptr=<value optimized out>, return_value_used=<value optimized out>) at /root/gearman-0.8.0/php_gearman.c:1937
#5 0x000000000081e788 in zend_do_fcall_common_helper_SPEC (execute_data=<value optimized out>)
    at /usr/src/debug/php-5.3.6/Zend/zend_vm_execute.h:316

[Taneli Leppä (rosmiitto)]

Yes, adding:

$gc->removeOptions(GEARMAN_CLIENT_FREE_TASKS);

fixes the issue.

[Taneli Leppä (rosmiitto)]

However, it causes other issues with the extension (such as PHP Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 140357493208336 bytes) in Unknown on line 0), downgrading to libgearman 0.20 works.

[Brian Aker (brianaker)]

Ok, thanks for finding this, let me poke at it.

On Jun 17, 2011, at 6:40 AM, smo wrote:

> However, it causes other issues with the extension (such as PHP Fatal
> error: Allowed memory size of 268435456 bytes exhausted (tried to
> allocate 140357493208336 bytes) in Unknown on line 0), downgrading to
> libgearman 0.20 works.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Released
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions

[Brian Aker (brianaker)]

Confirmed.

Thanks, I have a test case now.

On Jun 17, 2011, at 6:40 AM, smo wrote:

> However, it causes other issues with the extension (such as PHP Fatal
> error: Allowed memory size of 268435456 bytes exhausted (tried to
> allocate 140357493208336 bytes) in Unknown on line 0), downgrading to
> libgearman 0.20 works.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Released
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions

[Brian Aker (brianaker)]

Ok,... I have a test case now that checks this option (hopefully) throughout the API. The tree for this is currently lp:gearmand/build

If all passes I will push it to lp:gearmand (and hopefully package up a release).

Thanks!

On Jun 17, 2011, at 5:38 AM, smo wrote:

> Yes, adding:
>
> $gc->removeOptions(GEARMAN_CLIENT_FREE_TASKS);
>
> fixes the issue.
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Released
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions

[Taneli Leppä (rosmiitto)]

The fix seems to work, great thanks! Btw, you have the wrong date on Gearman.org (also you can put in my real name, Taneli Leppä, if you want to give credit) :)

[Brian Aker (brianaker)]

Thanks, and I've updated the changelog.

On Jun 20, 2011, at 1:51 AM, smo wrote:

> The fix seems to work, great thanks! Btw, you have the wrong date on
> Gearman.org (also you can put in my real name, Taneli Leppä, if you want
> to give credit) :)
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/785203
>
> Title:
> Assertion in libgearman/client.cc:1699 in latest trunk
>
> Status in Gearman Server and Client Libraries:
> Fix Committed
>
> Bug description:
> I'm getting this using php-gearman module (both 0.7.0 and 0.8.0)
> compiled against the latest libgearman trunk and gearmand:
>
> libgearman/client.cc:1669: void* _client_do(gearman_client_st*,
> gearman_command_t, const char*, size_t, gearman_unique_t&, const
> void*, size_t, size_t*, gearman_return_t*): Assertion
> `client->task_list' failed.
>
> Gearman 0.20 works fine though. Has there been an API change or is
> this a bug?
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/785203/+subscriptions