Add autopilot suite and code coverage gating to autopilot MPs

[Francis Ginther]

1) Add the autopilot test suite as a gating test for lp:autopilot MPs.
 - Begin with amd64 (and ideally i386) and add touch devices when the suite is ready. The QA team will notify CI when touch is ready to enable.
 - Note: we currently only have amd64 systems configured for this type of testing. Enabling tests on i386 is a stretch goal here.

2) Add code coverage as a gating test such that MPs which lower the code coverage are rejected.

1) is more urgent then 2).

[Thomi Richards (thomir-deactivatedaccount)]

The commands to generate coverage reports for the autopilot unit tests are as follows:

python -m coverage erase
python -m coverage run --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python -m coverage xml

The first command erases any existing coverage data. If you're running in a clean environment you don't need this (but it can't hurt).
The second command runs the tests, and generates coverage data in an internal data format.
The third and final command writes a file named 'coverage.xml' in the current directory that jenkins should be able to parse.

Let me know if you need anything else.

[Thomi Richards (thomir-deactivatedaccount)]

Actually, those commands should instead be:

python -m coverage erase
python -m coverage run --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python3 -m coverage run --append --branch --include "autopilot/*" -m autopilot.run run autopilot.tests.unit
python -m coverage xml --omit "autopilot/tests/*"

This will run the python3 unit test suite as well, and will omit test lines from the final coverage report.

Cheers,

[Francis Ginther (fginther)]

Attempts to add the autopilot suite to CI have run into:
https://bugs.launchpad.net/autopilot/+bug/1259721

The bug causes the test runner to hang. So we either need a fix or a special case workaround. A couple of fixes have been proposed, which look promising.

[Francis Ginther (fginther)]

QA team has requested adding the touch testing now (back around March 11). I attempted this here (this is the job chain):
http://s-jenkins.ubuntu-ci:8080/job/generic-mediumtests-touch/3265/
http://s-jenkins.ubuntu-ci:8080/job/generic-mediumtests-builder-saucy-armhf/3267/

This ultimately led to the autopilot armhf build that is attached, but I now see the problem. I linked in the wrong job (note the saucy above :-( )... I should have used generic-mediumtests-trusty-touch.