Problem in Drizzle plugin

[Andrew Hutchings]

In libgearman-server/plugins/queue/drizzle/query.cc the function drizzle_column_skip is used. Despite the description of this function this only skips one column and leaves the rest in network buffer for retrieval. This then breaks the row retrieval.

In libdrizzle I have just pushed up a fix for bug #528410 which adds the function drizzle_column_skip_all and also errors correctly if row retrieval is attempted before all columns have been retrieved.

For gearman I recommend calling drizzle_column_skip repeatedly until all columns have been retrieved (so loop drizzle_result_column_count() number of times). Note that this will still break in libdrizzle from Drizzle < 2011-08-01 due to bugs in drizzle_column_skip.

[Michael Hood (michaelhood)]

Corrected typo in Drizzle bug number (#548410 -> #528410).

[Andy Nemzek (andyn-j)]

Any plans on fixing this? I'm trying to use MySQL as a persistent queue for gearman, and can't really move forward till this is fixed. Would I have the same problem if I used Drizzle instead of MySQL? Or do you recommend any other DB's? Thanks!

[Andy Nemzek (andyn-j)]

FYI - I wrote more details on bug #528410, but basically this bug does not appear to exist in Drizzle7 2011.03.13. So, for those trying to get Gearman working with libdrizzle, just stay away from Drizzle7 2011.08.25 for now until this gets figured out.

[Andrew Hutchings (linuxjedi)]

All versions of Drizzle/libdrizzle are affected in different ways with the current gearman code. Sometimes it will be fine with 2011.03.13, other times you will get packet sequence errors or hangs.

[RutgerM (rutgerm)]

I have the same problem on Ubuntu 11.04

I installed both Gearman and Drizzle and no update in the table.

When I tested Drizzle in a way with Gearman I got some odd error which I can't reproduce because I recompiled the latest version.

I hope this is fixed soon as this is one of the best advantages!

[Oleg (ok-k)]

First of all thanks a lot for your job and commintent to this project. It's really useful to lots of people.

Concerning this bug - is there any estimations on dates of next release with a bugfix included?

[Brian Aker (brianaker)]

0.26 is up on the download servers right now, we just haven't sent out the announcement just yet.
gearmand-0.26.tar.gz

On Dec 4, 2011, at 1:35 PM, Oleg wrote:

> First of all thanks a lot for your job and commintent to this project.
> It's really useful to lots of people.
>
> Concerning this bug - is there any estimations on dates of next release
> with a bugfix included?
>
> --
> You received this bug notification because you are a bug assignee.
> https://bugs.launchpad.net/bugs/843782
>
> Title:
> Problem in Drizzle plugin
>
> Status in Gearman Server and Client Libraries:
> Fix Committed
>
> Bug description:
> In libgearman-server/plugins/queue/drizzle/query.cc the function
> drizzle_column_skip is used. Despite the description of this function
> this only skips one column and leaves the rest in network buffer for
> retrieval. This then breaks the row retrieval.
>
> In libdrizzle I have just pushed up a fix for bug #528410 which adds
> the function drizzle_column_skip_all and also errors correctly if row
> retrieval is attempted before all columns have been retrieved.
>
> For gearman I recommend calling drizzle_column_skip repeatedly until
> all columns have been retrieved (so loop drizzle_result_column_count()
> number of times). Note that this will still break in libdrizzle from
> Drizzle < 2011-08-01 due to bugs in drizzle_column_skip.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/gearmand/+bug/843782/+subscriptions

[Oleg (ok-k)]

Unfortunately it's not working...
I wasn't able to build gearmand - here's the output (with 0.25 i didn't get any errors):

  CXX libgearman-server/plugins/queue/drizzle/libgearman_server_libgearman_server_la-queue.lo
libgearman-server/plugins/queue/drizzle/queue.cc: In function 'gearmand_error_t gearman_server_queue_libdrizzle_init(gearmand::plugins::queue::Drizzle*, gearman_server_st*)':
libgearman-server/plugins/queue/drizzle/queue.cc:417: error: 'drizzle_column_skip_all' was not declared in this scope
make[1]: *** [libgearman-server/plugins/queue/drizzle/libgearman_server_libgearman_server_la-queue.lo] Error 1
make[1]: Leaving directory `/root/gearman/gearmand-0.26'
make: *** [all] Error 2

[Oleg (ok-k)]

Just for info, here're the configure options (it has worked fine with 0.25):

./configure --with-drizzled libdrizzle_CFLAGS="-I/usr/include/libdrizzle-2.0" libdrizzle_LIBS="-ldrizzle-2.0"

[Andrew Hutchings (linuxjedi)]

Oleg: you need an up to date version of libdrizzle too.

[Oleg (ok-k)]

Hmmmm
Actually i've got quite fresh ones installed...

libdrizzle-devel-2011.08.23-1.el5
libdrizzle-2011.08.23-1.el5

Although it seems there're a newer one available
http://launchpad.net/drizzle/fremont/2011-11-13/+download/drizzle7-2011.11.29.tar.gz
Is it the right one with fix included?
But unfortunately it's not available as rpm...

[Andrew Hutchings (linuxjedi)]

The Drizzle RPMs are a little old at the moment, not sure what the current state of the guy maintaining them is. The fix is in the source file you linked to.

[Oleg (ok-k)]

Problem is i wasn't able to build gearman and drizzle at the same server as drizzle requires boost 1.44 from drizzle-dev repo (i didn't have success to build gearman with that veriosn of boost).
So the only way is to wait until new version of rpm is built...

[Oleg (ok-k)]

Finally managed to solve the issue.
Compile string looks like this:

CXX=/usr/bin/g++44 CC=/usr/bin/gcc44 ./configure --with-boost=/usr/include/ --with-drizzled libdrizzle_CFLAGS="-I/opt/src/drizzle7-2011.11.29/libdrizzle-2.0" libdrizzle_LIBS="-ldrizzle-2.0"

Where /opt/src/drizzle7-2011.11.29 folder stores untared scr of drizzle.
Finally we get working persistent queue with Gearman 0.26 and Drizzle7-2011.11.29
(this works for CentOS)

[symphonyx (erich-beyrent)]

I wasn't able to get this to work on CentOS. When I run this, I get the following:

libgearman-server/plugins/queue/drizzle/queue.cc:24:39: error: libdrizzle/drizzle_client.h: No such file or directory
libgearman-server/plugins/queue/drizzle/queue.cc:46: error: 'drizzle_return_t' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:47: error: expected ',' or ';' before '{' token
libgearman-server/plugins/queue/drizzle/queue.cc:123: error: ISO C++ forbids declaration of 'drizzle_result_st' with no type
libgearman-server/plugins/queue/drizzle/queue.cc:123: error: expected ';' before '*' token
libgearman-server/plugins/queue/drizzle/queue.cc:128: error: expected ';' before 'void'
libgearman-server/plugins/queue/drizzle/queue.cc:148: error: ISO C++ forbids declaration of 'drizzle_st' with no type
libgearman-server/plugins/queue/drizzle/queue.cc:148: error: expected ';' before '*' token
libgearman-server/plugins/queue/drizzle/queue.cc:149: error: ISO C++ forbids declaration of 'drizzle_con_st' with no type
libgearman-server/plugins/queue/drizzle/queue.cc:149: error: expected ';' before '*' token
libgearman-server/plugins/queue/drizzle/queue.cc:150: error: ISO C++ forbids declaration of 'drizzle_con_st' with no type
libgearman-server/plugins/queue/drizzle/queue.cc:150: error: expected ';' before '*' token
libgearman-server/plugins/queue/drizzle/queue.cc:151: error: 'drizzle_result_st' does not name a type
libgearman-server/plugins/queue/drizzle/queue.cc: In constructor 'gearmand::plugins::queue::Drizzle::Drizzle()':
libgearman-server/plugins/queue/drizzle/queue.cc:170: error: class 'gearmand::plugins::queue::Drizzle' does not have any field named 'drizzle'
libgearman-server/plugins/queue/drizzle/queue.cc:171: error: class 'gearmand::plugins::queue::Drizzle' does not have any field named 'con'
libgearman-server/plugins/queue/drizzle/queue.cc:172: error: class 'gearmand::plugins::queue::Drizzle' does not have any field named 'insert_con'
libgearman-server/plugins/queue/drizzle/queue.cc:173: error: class 'gearmand::plugins::queue::Drizzle' does not have any field named '_result'
libgearman-server/plugins/queue/drizzle/queue.cc:189: error: 'drizzle' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:189: error: 'drizzle_create' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:190: error: 'con' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:190: error: 'drizzle_con_create' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:191: error: 'insert_con' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc: In destructor 'virtual gearmand::plugins::queue::Drizzle::~Drizzle()':
libgearman-server/plugins/queue/drizzle/queue.cc:196: error: 'con' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:196: error: 'drizzle_con_free' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:197: error: 'insert_con' was not declared in this scope
libgearman-server/plugins/queue/drizzle/queue.cc:198: error: 'drizzle' was not declared in this scope
l...