Python Gearman worker intermittently stop receiving jobs from gearmand
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Gearman |
New
|
Undecided
|
Brian Aker |
Bug Description
I found that when using multiple gearmand servers and having the workers connect to both of them, somehow (intermittently) some of the workers will just accept jobs from one of the servers only. From the log I created, it seems that gearmand stops sending job offer to those workers. This causes the jobs from that server to be executed only by the workers which are still connected to that server. I experienced this in a 16-core Ubuntu machine.
Configuration:
1. Start gearmand at ports 4730 and 4731
2. Start worker B, C, D, and E, all connect to both 4730 and 4731
3. Start client A, connects to both 4730 and 4731
4. After many jobs are sent, some of the workers will randomly stop receiving job offers from one of the servers.
Try the minimal reproducing code as attached below to test.
1. Start gearmand at 4730 and 4731
2. Start gearman_worker.py
3. Start test.bash
4. Check that for the last n jobs (as seen in worker.log), only some of the workers are processing the requests (i.e., some others just don't fetch jobs from the other server anymore). Note that this happens intermittently, so please rerun the worker and the bash script if the behaviour hasn't occur yet.
I used gearmand 1.1.9 on 16-core Ubuntu 11.04
The worker and client are using Python 2.7 with python-gearman version 2.0.2
Attached is the zip of logfiles.
Note that starting from line 40969 onwards, there are only two workers which receive jobs from server 4730 (worker 00 and worker 01). Also note that the other workers didn't lose connection with server, as we can see at lines 40970 and 40971 that the other two workers are waiting with connections open to both servers.
I modified the connection_
By analyzing the code in connection_
Expected behaviour:
Worker will always receive jobs from any server that still has jobs available.
summary: |
- Gearman worker intermittently stop receiving jobs from gearmand + Python Gearman worker intermittently stop receiving jobs from gearmand |
Thank you! Investigating.