gearmand high cpu after too large job

[Steven Barre]

Our gearmand server started going into a state where it used a lot of CPU.

watching the general log in mysql it seemed to be stuck in a loop running

Prepare INSERT INTO gearman_1_queue (unique_key, function_name, priority, data, when_to_run) VALUES(?, ?, ?, ?, ?)

running with --verbose DEBUG didn't seem to indicate anything, and straceing the main pid showed no activity.

After much digging I found the issue was with max_allowed_packet in mysql. If we submitted a background job that was too big the server would go into this state. Increasing this value fixed our problem.

gearmand should handle this better and return an error to the job submitter instead of going into infinite loop.

# uname -r
2.6.32-431.29.2.el6.x86_64

# cat /etc/redhat-release
CentOS release 6.5 (Final)

# rpm -qa gearmand
gearmand-1.1.8-2.el6.x86_64

# gearmand --version
gearmand 1.1.8 - https://bugs.launchpad.net/gearmand

# cat /etc/sysconfig/gearmand
OPTIONS="--queue-type=MySQL --mysql-host=localhost --mysql-port=3306 --mysql-user=gearman --mysql-password=XXXX --mysql-db=gearman --mysql-table=gearman_1_queue --round-robin"