Jobs generated by jenkins job builder are not registered

[Antoine "hashar" Musso]

When creating jobs with Jenkins job builder, they are created and runnable in Jenkins but the new job are not registered with the Gearman server.

My setup is:

- a single Jenkins master (with several slaves)
- LTS version 1.565.1 (1.554.x serie was affected as well)
- Gearman Plugin 0.0.7 from Jenkins update center

I use Jenkins Job Builder from master.

Querying Gearman server using the 'worker' and 'status' commands yield no result for the jobs created. Triggering a change in Gerrit ends up with Zuul complaining about the job not being registered (not surprising).

Jenkins log shows nothing, but then I am not sure I have proper log setup for the Gearman plugin (any tip welcome).

From the access log:

    [07/Aug/2014:08:05:06 +0000] "GET /ci/job/mwext-Vine-lint/api/json HTTP/1.1" 404 1396 "-" "Python-urllib/2.7"
    [07/Aug/2014:08:05:06 +0000] "GET /ci/job/mwext-Vine-lint/api/json HTTP/1.1" 404 1396 "-" "Python-urllib/2.7"
    [07/Aug/2014:08:05:07 +0000] "GET /ci/job/mwext-Vine-lint/api/json HTTP/1.1" 404 1396 "-" "Python-urllib/2.7"
    [07/Aug/2014:08:05:07 +0000] "POST /ci/createItem HTTP/1.1" 200 0 "-" "Python-urllib/2.7"
    [07/Aug/2014:08:05:08 +0000] "GET /ci/job/mwext-Vine-lint/api/json HTTP/1.1" 200 26 "-" "Python-urllib/2.7"

I suspect the Jenkins API createItem uses a slightly different code path than the one used to save a configuration from the web interface.

If I head to the job configuration page and save the config. That triggers a re-registration of all the jobs and the job shows up in the 'status':

    $ echo status|nc -q 2 localhost 4730|fgrep mwext-Vine-lint
    build:mwext-Vine-lint 0 0 11
    build:mwext-Vine-lint:UbuntuPrecise 0 0 11
    build:mwext-Vine-lint:hasSlaveScripts 0 0 11
    $

[Khai Do (zaro0508)]

Openstack is still using the gearman plugin with Jenkins LTS 1.532.2 so not surprising that it's not working with later versions. My guess is that the Jenkins events have changed somehow, maybe an event that the gearman plugin relies on is no longer being fired :(

Instructions on how to setup Logging:
1. goto http://host:8080/log/levels
2. add "org.gearman.session.logger" with level "WARNING"
3. goto http://host:8080/log/all
now you should see logs from gearman plugin.

[Antoine "hashar" Musso (hashar)]

Thank you very much for the step by step instruction. I have enabled the log, will look at them whenever the issue occur again and post them here.

[Antoine "hashar" Musso (hashar)]

Our bug before I forget about it : https://bugzilla.wikimedia.org/show_bug.cgi?id=63758

[Antoine "hashar" Musso (hashar)]

I have setup a Jenkins logger using:

 org.gearman.session.logger
 WARNING

After a little while, I have some warning messages such as:

Aug 17, 2014 6:41:59 PM WARNING org.gearman.common.GearmanJobServerSession closeSession
Attempted to close a session that is not open: GearmanJobServerSession:284:GearmanNIOJobServerConnection:localhost/127.0.0.1:4730

Might be the Gearman plugin losing its connection with Zuul Gearman embed server, or it might occurs when I manually disable / reenable the Gearman connection.

No other kind of messages are shown.

[Antoine "hashar" Musso (hashar)]

I had the issue when creating a new job via the web interface. Apparently the Zuul gearman server stopped responding (forked process is still running but TCP connections to it timeout). Jenkins ends up using a bunch of CPU waiting for the port and does not process anything else.

Work around:
* disable the Gearman connection
* save config

The Zuul gearman server is happy again as shown with a workers command:

13 208.80.154.135 - :
14 208.80.154.135 Zuul Merger : merger:merge merger:update
82 208.80.154.135 Zuul RPC Listener : zuul:get_running_jobs zuul:promote zuul:enqueue
110 208.80.154.135 - :
16 127.0.0.1 - :
.

The status command still report all the jobs albeit with 0 workers available.

* reenable gearman connection
* save config

The workers are repopulated and status shows jobs have workers available.

Maybe that is a bug in the python gear module.

[Antoine "hashar" Musso (hashar)]

I have give it a try again today.

In the Zuul gearman logging configuration I created a new DEBUG logger for the qualifier 'gear.Server'. That class has interesting debugging messages such as packet handling, registration of functions and reset of all functions.

------------

Test 1 - web interface:

I have clicked the 'New item' link, create a new freestyle project with the name 'MyJob'

The Gearman plugin did send a packet to reset all functions and properly registered a function 'build:MyJob'.

I am being shown the form to configure the job. Click save, all functions are reset and reregistered. 'build:MyJob' is properly registered with whatever labels I might have come up with.

Test 2 - JJB

I create a dummy job named 'MyJJBJob'

In the web interface the job is shown.

No packet have been sent by the Gearman plugin.

------------

With my limited Java and Jenkins API knowledge, the the Gearman plugin has an implementation of 'SaveableListener' ( http://javadoc.jenkins-ci.org/hudson/model/listeners/SaveableListener.html ) which would react on save. I guess that event is emitted via the web form but not on creation of item via the API.

There is another one 'ItemListener' which might be triggered on creation of jobs via the rest API: http://javadoc.jenkins-ci.org/hudson/model/listeners/ItemListener.html

[Khai Do (zaro0508)]

Thanks for the additional info. It was very helpful. I'm not sure if the ItemListener will work because jjb creates 'projects' in jenkins. IIRC Jenkins 'jobs' aren't necessarily the same as 'projects'. I believe the term 'job' actually means the same thing as a 'build' as in a 'build job'. I will look into it thought.

[Antoine "hashar" Musso (hashar)]

Another way would be to have Jenkins core to emit SaveableListener events whenever a job is created via the config.xml API :-)

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/125755
Committed: https://git.openstack.org/cgit/openstack-infra/gearman-plugin/commit/?id=6307ba2225bbc9e8d4f41062bd6b944e40d7fa7a
Submitter: Jenkins
Branch: master

commit 6307ba2225bbc9e8d4f41062bd6b944e40d7fa7a
Author: Khai Do <email address hidden>
Date: Thu Oct 2 12:44:49 2014 -0700

    update to work with Jenkins LTS ver 1.565.3

    This change updates the gearman-plugin to build against Jenkins LTS
    ver 1.565.3. This is required to support later versions of Jenkins
    because there have been changes to Jenkins events in core.

    This change adds the ItemListener to re-register gearman
    functions on changes to jenkins projects. Using the ItemListener is
    better because it provides more details on the item that's been
    changed and the events are more grandular. There was also a change
    to ComputerListener events (onTemporaryOffline and onTemporaryOnline)
    that needed to be handled seperately in this change. The SaveableListener
    is still needed due to a bug in ItemListener.

    The most notable change to function registration is that the gearman
    plugin will no longer register functions containing a node's self label.

    For example:
    Assume you have the following setup:
      a node: named 'trusty-slave1' and labeled 'trusty'
      a job: named 'my-job' that targets the 'trusty' label

    The gearman plugin used to register the following functions:
    'build:my-job', 'build:my-job:trusty' and
    'build:my-job:trusty-slave1'

    With this update the gearman plugin will only register
    'build:my-job' and 'build:my-job:trusty'. It will no
    longer register functions containing the implicit
    node name (trusty-slave1).

    If your gearman client has been using explicit labels
    to execute builds then this change will not affect
    your workflow.

    Closes-Bug: #1353891
    Change-Id: I9e57ec9f24bf303989a4df1fc4f1a0c4b6d001bc

[Antoine "hashar" Musso (hashar)]

That solved the issue for me.

v0.1.0 includes the fix to properly register jobs when created over the API
v0.1.1 fix job results which were not being sent back to gearman client

Thank you Khai!