configure separate coverage test job in jenkins & tox

[Doug Hellmann]

We need to set up tox to run a separate environment called "cover" to get the coverage stats. The job should emit a coverage.xml file in the root of the project. The CI team can then fix up the ceilometer-coverage job in Jenkins to pick up that file and give us pretty graphs.

Background:

clarkb: is there a way to kick jobs out of the unstable jobs list in jenkins?
[4:40pm] clarkb: ceilometer-docs and -coverage are no longer run anymore and I don't like noise
[4:43pm] jeblair: clarkb: if they aren't run anymore, i think it's okay to delete them.
[4:44pm] dhellmann: is ceilometer-coverage meant to be set up as separate from the actual test run?
[4:44pm] dhellmann: we're using readthedocs.org for hosting docs, so we don't need ceilometer-docs
[4:44pm] clarkb: dhellmann: yes, for other projects it is run as a post merge job
[4:45pm] clarkb: yup the jobs were created because they are part of a template and were added to zuul so they got run once then were remvoed from zuul
[4:45pm] clarkb: basically all is well except for the unstable jobs on my dashboard
[4:45pm] clarkb: jeblair: jenkins jobs will just readd the job I think, but that is ok
[4:45pm] mtaylor: clarkb: ++
[4:46pm] mtaylor: clarkb: yeah, just delete it, zull won't re-kick, therefore, no more unstable jobs
[4:46pm] mtaylor: clarkb: if you wnated to - you could add a skip-list to the template processing in job filler
[4:46pm] dhellmann: what does the coverage job do that running the tests with --with-coverage does not do?
[4:46pm] • dhellmann wonders if he is missing an opportunity for a pretty graph
[4:46pm] mtaylor: dhellmann: so - in general
[4:47pm] mtaylor: dhellmann: we do not use run_test.sh in any way in the jenkins setup
[4:47pm] clarkb: the button in the gooey says "delete project" in this case project == job ?
[4:47pm] mtaylor: dhellmann: BUT - if your tests actually work with coverage...
[4:47pm] mtaylor: clarkb: why is the ceilometer-coverage job busted?
[4:47pm] dhellmann: mtaylor, the tests do work with coverage
[4:48pm] mtaylor: dhellmann: great - then you're missing the opportunity to have pretty graphs
[4:48pm] dhellmann: and we have tox configured to run run_tests.sh because our wrapper deletes the file from the virtualenv that prevents it from seeing the system site packages (giving us access to qpid libs)
[4:48pm] • clarkb looks more closely at the coverage job
[4:48pm] clarkb: https://jenkins.openstack.org/job/ceilometer-coverage/1/console
[4:48pm] dhellmann: mtaylor, clarkb: it is very likely a failure to configure something on our part
[4:49pm] mtaylor: dhellmann: ah. WELL - that should actually be fixable in straight tox now
[4:49pm] clarkb: the first and only time it ran it looks like tox was missing a config item
[4:49pm] clarkb: then it was removed from zuul's layout.conf
[4:49pm] dhellmann: mtaylor, yeah, that was always meant to be a temporary hack
[4:49pm] mtaylor: dhellmann: add "sitepackages = True" to each testenv section
[4:49pm] mtaylor: and you can remove the file hack
[4:50pm] dhellmann: mtaylor, that worked on the py27 system, but not the py26 boxes
[4:50pm] mtaylor: dhellmann: and then if your tox coverage environment works and produces a coverage.xml file in the root dir, we can start running it again
[4:50pm] dhellmann: were those boxes updated to the latest tox & virtualenv?
[4:50pm] mtaylor: dhellmann: good question...
[4:50pm] mtaylor: looking
[4:50pm] mtaylor: might be virtualenv version ...
[4:50pm] mtaylor: on the oneiric boxes
[4:50pm] mtaylor: dhellmann: no matter - orthogonal problem
[4:50pm] dhellmann: mtaylor: I'll add setting up the coverage job to our todo list
[4:51pm] mtaylor: dhellmann: make sure that the coverage thing works and produces coverage.xml and we can hook you up
[4:51pm] dhellmann: mtaylor: got it
[4:51pm] • mtaylor needs to document pre-conditions and expected outcome of out standard tox envs...
[4:51pm] clarkb: but the -docs job will be deleted
[4:51pm] dhellmann: is it acceptable to paste this conversation into the bug report as an explanation?
[4:51pm] mtaylor: yah
[4:51pm] dhellmann: clarkb: yes, please feel free to delete the -docs job
[4:52pm] mtaylor: totally

[Justin Shepherd (jshepher)]

Looks like this bug was resolved by https://review.openstack.org/#/c/10977/