a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used

Bug #480775 reported by Dave Webb on 2009-11-11
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gearman
High
Eric Day

Bug Description

When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.

I'm running gearmand 0.10 with a command like the following:

/usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
        --log-file=/logs/local_gearman.log \
        --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
        --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v

I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().

Looking at the gearmand logs I see lines like:

DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
DEBUG libdrizzle add:
CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
 INFO [ 0] :::43190 Disconnected

This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.

I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.

I'm using libdrizzle-0.4 if that makes a difference.

Any ideas?

Dave.

summary: - gearmand fails to add background job when mysql persistant queues are
- used
+ a blank unique_key means gearmand fails to add any background jobs when
+ mysql persistent queues are used

Looking further into the source, libgearman-server/queue_libdrizzle.c has the code that actually builds up the SQL (starting at line 369).

Eric Day (eday) wrote :

The easy workaround is to specify unique keys in your job submission. This is a good idea anyways to take advantage of job coalescing. Having said that, this can still be fixed to automatically generate a UUID when a job is submitted with no unique ID. This is a trivial fix and will be on the todo for next release.

Changed in gearmand:
status: New → Confirmed
milestone: none → 0.11
assignee: nobody → Eric Day (eday)
importance: Undecided → High

Thanks for looking into this Eric.

The perl client (Gearman::Client) doesn't seem to support providing unique keys. I've tried passing in a Gearman::Task object as well as passing in the function name and arguments separately.

I was assuming that the unique_key in the database was in some way generated by the server based upon the function name and an internal counter or something like that.

Gearman::Client does support unique keys, it's the 'uniq' option to
tasks (options hash for do calls or argument to add_task). There are
some examples in the perldocs on how to specifiy it.

On Wed, Nov 11, 2009 at 07:22:09PM -0000, Dave Webb wrote:
> Thanks for looking into this Eric.
>
> The perl client (Gearman::Client) doesn't seem to support providing
> unique keys. I've tried passing in a Gearman::Task object as well as
> passing in the function name and arguments separately.
>
> I was assuming that the unique_key in the database was in some way
> generated by the server based upon the function name and an internal
> counter or something like that.
>
> --
> a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used
> https://bugs.launchpad.net/bugs/480775
> You received this bug notification because you are a member of Gearman-
> developers, which is the registrant for Gearman C Server and Library.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.
>
> I'm running gearmand 0.10 with a command like the following:
>
> /usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
> --log-file=/logs/local_gearman.log \
> --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
> --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v
>
> I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().
>
> Looking at the gearmand logs I see lines like:
>
> DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
> DEBUG libdrizzle add:
> CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
> FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
> INFO [ 0] :::43190 Disconnected
>
> This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.
>
> I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.
>
> I'm using libdrizzle-0.4 if that makes a difference.
>
> Any ideas?
>
> Dave.

Unfortunately I'm using the fire-and-forget dispatch_background() method, and that doesn't seem to have a unique-key option.

gearmand doesn't appear to use the persistent storage for jobs where the client waits for a result. And these are the only jobs where you can pass in a explicit unique key (using the perl client anyway).

Eric Day (eday) wrote :

No, gearmand doesn't store foreground jobs persistently because the
connected client can failover if something should happen. Persistence
really only makes sense for background jobs.

You can pass a unique key for background jobs with Gearman::Client:

use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
$ref= $client->dispatch_background('reverse', 'Hello World', {'uniq' => 'my_unique_key'});
print "$$ref\n";

-Eric

On Wed, Nov 11, 2009 at 08:20:27PM -0000, Dave Webb wrote:
> gearmand doesn't appear to use the persistent storage for jobs where the
> client waits for a result. And these are the only jobs where you can
> pass in a explicit unique key (using the perl client anyway).
>
> --
> a blank unique_key means gearmand fails to add any background jobs when mysql persistent queues are used
> https://bugs.launchpad.net/bugs/480775
> You received this bug notification because you are a member of Gearman-
> developers, which is the registrant for Gearman C Server and Library.
>
> Status in Gearman C Server and Library: Confirmed
>
> Bug description:
> When running gearmand backed onto a mysql database for queue persistence, only one job of any type can be added. This appears to be because the SQL used to insert the jobs has a blank value for "unique_key" which is the primary key of the queue table.
>
> I'm running gearmand 0.10 with a command like the following:
>
> /usr/local/sbin/gearmand --daemon --port=4730 --user=gearmand \
> --log-file=/logs/local_gearman.log \
> --queue-type=libdrizzle --libdrizzle-mysql --libdrizzle-user=gearman \
> --libdrizzle-db=gearman_queues --libdrizzle-table=gearman_queue1 -v -v -v
>
> I do not get a job handle back when calling the perl client with Gearman::Client->dispatch_background().
>
> Looking at the gearmand logs I see lines like:
>
> DEBUG [ 0] :::43190 Received SUBMIT_JOB_BG
> DEBUG libdrizzle add:
> CRAZY libdrizzle query: INSERT INTO gearman_queue1 SET priority=1,unique_key='',function_name='my_function',data='^E^F^C\
> FATAL _libdrizzle_query:drizzle_query:Duplicate entry '' for key 'PRIMARY'
> INFO [ 0] :::43190 Disconnected
>
> This seems to indicate that the SQL being build up by the gearmand has a blank value for unique_key. This causes all jobs subsequent to the first to fail to be added.
>
> I don't program in c, but the problem seems to be in the libgearman-server/job.c file, where the SQL is assembled.
>
> I'm using libdrizzle-0.4 if that makes a difference.
>
> Any ideas?
>
> Dave.

Doh! Sorry I totally missed the uniq param in the Gearman::Task sourcecode. I added a GUID for the uniq arg, and it all seems to work.

Thanks again Eric.

Eric Day (eday) wrote :

Due to the compatibility with some of the existing modules and the Perl server, I think it may be best to just require people to supply a unique ID with persistent background jobs. I started auto-generating unique ID's for those that did not, but could see a few cases where this could cause ambiguities and not coalesce. Marking this bug as invalid since the workaround is trivial and encourages a best practice.

Changed in gearmand:
status: Confirmed → Won't Fix
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers