a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used

Bug #480775 reported by Dave Webb
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gearman
Won't Fix
High
Eric Day

Bug Description

When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.

I'm running gearmand 0.10 with a command like the following:

/usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
        --log-file=/logs/local_gearman.log \
        --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
        --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v

I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().

Looking at the gearmand logs I see lines like:

DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
DEBUG libdrizzle add:
CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
 INFO [ 0] :::43190 Disconnected

This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.

I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.

I'm using libdrizzle-0.4 if that makes a difference.

Any ideas?

Dave.

summary: - gearmand fails to add background job when mysql persistant queues are
- used
+ a blank unique_key means gearmand fails to add any background jobs when
+ mysql persistent queues are used
Revision history for this message
Dave Webb (x-launchpad-net-dcw-co-nz) wrote :

Looking further into the source, libgearman-server/queue_libdrizzle.c has the code that actually builds up the SQL (starting at line 369).

Revision history for this message
Eric Day (eday) wrote :

The easy workaround is to specify unique keys in your job submission. This is a good idea anyways to take advantage of job coalescing. Having said that, this can still be fixed to automatically generate a UUID when a job is submitted with no unique ID. This is a trivial fix and will be on the todo for next release.

Changed in gearmand:
status: New → Confirmed
milestone: none → 0.11
assignee: nobody → Eric Day (eday)
importance: Undecided → High
Revision history for this message
Dave Webb (x-launchpad-net-dcw-co-nz) wrote :

Thanks for looking into this Eric.

The perl client (Gearman::Client) doesn't seem to support providing unique keys. I've tried passing in a Gearman::Task object as well as passing in the function name and arguments separately.

I was assuming that the unique_key in the database was in some way generated by the server based upon the function name and an internal counter or something like that.

Revision history for this message
Eric Day (eday) wrote : Re: [Bug 480775] Re: a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used

Gearman::Client does support unique keys, it's the 'uniq' option to
tasks (options hash for do calls or argument to add_task). There are
some examples in the perldocs on how to specifiy it.

On Wed, Nov 11, 2009 at 07:22:09PM -0000, Dave Webb wrote:
> Thanks for looking into this Eric.
>
> The perl client (Gearman::Client) doesn't seem to support providing
> unique keys. I've tried passing in a Gearman::Task object as well as
> passing in the function name and arguments separately.
>
> I was assuming that the unique_key in the database was in some way
> generated by the server based upon the function name and an internal
> counter or something like that.
>
> --
> a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used
> https://bugs.launchpad.net/bugs/480775
> You received this bug notification because you are a member of Gearman-
> developers, which is the registrant for Gearman C Server and Library.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.
>
> I'm running gearmand 0.10 with a command like the following:
>
> /usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
> --log-file=/logs/local_gearman.log \
> --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
> --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v
>
> I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().
>
> Looking at the gearmand logs I see lines like:
>
> DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
> DEBUG libdrizzle add:
> CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
> FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
> INFO [ 0] :::43190 Disconnected
>
> This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.
>
> I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.
>
> I'm using libdrizzle-0.4 if that makes a difference.
>
> Any ideas?
>
> Dave.

Revision history for this message
Dave Webb (x-launchpad-net-dcw-co-nz) wrote :

Unfortunately I'm using the fire-and-forget dispatch_background() method, and that doesn't seem to have a unique-key option.

Revision history for this message
Dave Webb (x-launchpad-net-dcw-co-nz) wrote :

gearmand doesn't appear to use the persistent storage for jobs where the client waits for a result. And these are the only jobs where you can pass in a explicit unique key (using the perl client anyway).

Revision history for this message
Eric Day (eday) wrote :

No, gearmand doesn't store foreground jobs persistently because the
connected client can failover if something should happen. Persistence
really only makes sense for background jobs.

You can pass a unique key for background jobs with Gearman::Client:

use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
$ref= $client->dispatch_background('reverse', 'Hello World', {'uniq' => 'my_unique_key'});
print "$$ref\n";

-Eric

On Wed, Nov 11, 2009 at 08:20:27PM -0000, Dave Webb wrote:
> gearmand doesn't appear to use the persistent storage for jobs where the
> client waits for a result. And these are the only jobs where you can
> pass in a explicit unique key (using the perl client anyway).
>
> --
> a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used
> https://bugs.launchpad.net/bugs/480775
> You received this bug notification because you are a member of Gearman-
> developers, which is the registrant for Gearman C Server and Library.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.
>
> I'm running gearmand 0.10 with a command like the following:
>
> /usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
> --log-file=/logs/local_gearman.log \
> --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
> --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v
>
> I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().
>
> Looking at the gearmand logs I see lines like:
>
> DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
> DEBUG libdrizzle add:
> CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
> FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
> INFO [ 0] :::43190 Disconnected
>
> This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.
>
> I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.
>
> I'm using libdrizzle-0.4 if that makes a difference.
>
> Any ideas?
>
> Dave.

Revision history for this message
Dave Webb (x-launchpad-net-dcw-co-nz) wrote :

Doh! Sorry I totally missed the uniq param in the Gearman::Task sourcecode. I added a GUID for the uniq arg, and it all seems to work.

Thanks again Eric.

Revision history for this message
Eric Day (eday) wrote :

Due to the compatibility with some of the existing modules and the Perl server, I think it may be best to just require people to supply a unique ID with persistent background jobs. I started auto-generating unique ID's for those that did not, but could see a few cases where this could cause ambiguities and not coalesce. Marking this bug as invalid since the workaround is trivial and encourages a best practice.

Changed in gearmand:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.