Update command not creating jobs

[Michael Loo]

Hi There,

Specs:
- Mac OSX 10.9.2 (Virtual Machine = VM)
  - jenkins-job-builder v0.8.0 (downloaded with brew)
  - Jenkins CI version 1.574
  - Python 2.7

- Mac OSX 10.9.4 (Physical Machine = PM)
  - jenkins-job-builder v0.7.0
  - Python 2.7

So I noticed an strange but annoying bug when trying to update jobs from the virtual machine.
When executing the same command on the VM and the PM, it produces 2 different outcomes:

1. The output below comes from executing jenkins-job-builder on the VM:

    vm_user$ jenkins-jobs --conf ./jenkins_jobs.ini delete job_name
    INFO:root:Deleting jobs in [job_name]
    INFO:jenkins_jobs.builder:Deleting jenkins job job_name
    vm_user$ jenkins-jobs --conf ./jenkins_jobs.ini update ./
    INFO:root:Updating jobs in ./ ([])

As you can see in the command above, no changes were applied although I just deleted the job

2. Now the second output below comes from executing jenkins-job-builder on the PM:

    pm_user$ jenkins-jobs --conf ./jenkins_jobs.ini delete job_name
    INFO:root:Deleting jobs in [job_name]
    INFO:jenkins_jobs.builder:Deleting jenkins job job_name
    pm_user$ jenkins-jobs --conf ./jenkins_jobs.ini update ./
    INFO:root:Updating jobs in ./ ([])
    INFO:jenkins_jobs.builder:Creating jenkins job job_name

And here, we can see that the job has been re-created as expected.

-----

I tried the following in order to debug what was going wrong:
- Reverting back to a version of the templates I am SURE worked (tested and committed months ago).
- Making sure the config had the right values (i.e.: password and cached ignored)
- Deleting the cache manually on the VM where Jenkins is being hosted on (also where the bug is occurring)
- Launching the job from a Jenkins' job with the same config and commands.

In all of the above I was able to reproduce the bug.
Then the last option I had was to check the version of jenkins-job-builder (which was hard to find**) on both machine and the VM had 0.8.0 while the other had 0.7.0.

Reverting back the version on the VM from 0.8.0 to 0.7.0 worked and the bug did not occur anymore!

In other words, the bug happens only on the version 0.8.0, so would you please look into it?

If you need more information, let me know.

** Would it be possible to add a `--version` or `-v` command to get the version?

Best Regards,

Michael Loo

[Clark Boylan (cboylan)]

I have no way of testing OS X or brew on OS X so this is hard for me to triage. However, if you can reproduce this using pip installed JJB that will at least get us closer to making this a reproduceable bug.

To do this:

install virtualenv
virtualenv venv
source venv/bin/activate
pip install jenkins-job-builder
run your test as above
pip freeze | grep jenkins # will show you the version of JJB installed
deactivate # this will exit the python virtualenv environment

This will install the latest version of JJB and python-jenkins which gives us a baseline for digging into what may be broken.

[Khai Do (zaro0508)]

There was a bug with the delete job command. The fix is here: https://review.openstack.org/#/c/127370/

Good idea about the --version flag. Could you please enter a separate feature request in launchpad for that?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/127844

[Khai Do (zaro0508)]

Version flag has been implemented in change: https://review.openstack.org/#/c/127844/

[Khai Do (zaro0508)]

Oh one last thing. The update command works fine for me. Might not have worked for you because the job didn't actually get deleted before the update because there was a bug with the delete command?

[Darragh Bailey (dbailey-k)]

hmm, think this problem pre-dates the recent delete bug.

The only way I could replicate this problem was to trash the entry-points.txt file, since JJB doesn't generate an exception when no entry points are found to generate the XML.

I suspect that downgrading resulted in correctly creating the needed egg files if using pip directly, or that the needed files were deleted by accident.

If you were to upgrade to 0.8.0 and test again, does the problem persist? If so, can you provide info on what package you are using or how you are downgrading/upgrading.

[Michael Loo (michael-loo)]

Hi, many thanks for looking into this. You guys are the best :)

Unfortunately I'm caught up with over works at the moment and won't be able to test the solution implemented in the master branch.

To answer some of your questions:

>> Might not have worked for you because the job didn't actually get deleted before the update because there was a bug with the delete command?
That might have been the case, but there was no error message whatsoever when deleting or updating as you can see in the output in the bug's description.

A theory I had was that the jobs generated under different versions of the software would result in an 'unclean' state where the jobs are not deleted properly which causes the new version of JJB to think that it is still present.
Did we delete function get updated in any way?

>> If you were to upgrade to 0.8.0 and test again, does the problem persist?
Yes it persisted, that was one of the tests I did a few months ago.

>> If so, can you provide info on what package you are using or how you are downgrading/upgrading.
Like I mentioned, I won't be able to do that before next week unfortunately.

[Michael Loo (michael-loo)]

I will look into reproducing this further early next week.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127844
Committed: https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/commit/?id=b77a9991eed45a31dea7941ed0acf5935dc3869e
Submitter: Jenkins
Branch: master

commit b77a9991eed45a31dea7941ed0acf5935dc3869e
Author: Khai Do <email address hidden>
Date: Sun Oct 12 09:49:13 2014 -0700

    Add a --version command line flag to show JJB version

    Closes-Bug: #1349634
    Change-Id: I4c9af2fa61383585467554272b04c5c4dba1568e

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/127370
Committed: https://git.openstack.org/cgit/openstack-infra/jenkins-job-builder/commit/?id=b7ff37ca5dfa1e44447387c636b8b0108404a0bf
Submitter: Jenkins
Branch: master

commit b7ff37ca5dfa1e44447387c636b8b0108404a0bf
Author: Khai Do <email address hidden>
Date: Thu Oct 9 15:08:18 2014 -0700

    fix delete job command and add tests

    change I126751e3 introduced recursive file definition feature however it broke
    the ability to delete jobs. This changes fixes it and adds a few tests for it.

    This was the error when trying to delete a job:

    (py27)~/jenkins-job-builder$ jenkins-jobs --conf jenkins_jobs.ini delete myjob
    INFO:root:Deleting jobs in [myjob]
    Traceback (most recent call last):
      File "/jenkins-job-builder/.tox/py27/bin/jenkins-jobs", line 10, in <module>
        sys.exit(main())
      File "/jenkins-job-builder/jenkins_jobs/cmd.py", line 122, in main
        execute(options, config)
      File "/jenkins-job-builder/jenkins_jobs/cmd.py", line 207, in execute
        builder.delete_job(job, options.path)
      File "/jenkins-job-builder/jenkins_jobs/builder.py", line 611, in delete_job
        self.load_files(fn)
      File "/jenkins-job-builder/jenkins_jobs/builder.py", line 576, in load_files
        if os.path.isdir(path):
      File "/jenkins-job-builder/.tox/py27/lib/python2.7/genericpath.py", line 41,
        in isdir st = os.stat(s)
    TypeError: coercing to Unicode: need string or buffer, NoneType found

    Closes-Bug: #1349634
    Change-Id: Ib87fa497d80ba33470c049f875658a3878afb664