Need a paralellized jenkins job that runs autopilot suites on silos

[Michał Sawicz]

We need something similar to:

http://q-jenkins.ubuntu-ci:8080/job/autopilot-release-gatekeeper/

but available to other projects. Running autopilot tests manually as part of test plans is a waste of time, which we never have enough...

To run through all the projects, the job takes over 3 hours, running all suites in sequence on one device, when they could be ran in parallel on $num_devices that are available at the moment, and cut the time by ~that number.

[Francis Ginther (fginther)]

I had further follow-up with Saviq on #ubuntu-ci-eng.

In summary:
 * A job that runs everything serially would work as a first step (better then nothing)
 * Installed image should be configurable (to a channel or number) and default to devel-proposed
 * Specifying the set of binaries to install from the silo is ok (believed to be a better option then install all of them or doing a dist-upgrade).

<fginther> Saviq, is a job that runs all the ap suites in serial of any use?
<Saviq> fginther, sure
<Saviq> fginther, would be a stop-gap, but still faster than running them manually
<Saviq> fginther, and, fwiw, more reliable
<fginther> Saviq, what would the image baseline need to be? proposed?
<Saviq> fginther, that'd probably be used most often
<Saviq> fginther, but since it's flashing every time anyway, would that really matter?
<Saviq> fginther, should be possible to flash with any channel / revision even
<fginther> Saviq, I guess I meant proposed vs stable, etc.
<Saviq> fginther, in terms of system image, not distro, right?
<fginther> Saviq, yes, I'm referreing to the system image, my terminology may be off
<Saviq> fginther, but yeah, same applies, since the devices are being flashed every time anyway, it should be possible to select which channel / revision to flash
<Saviq> fginther, channel = devel-proposed, revision = 283 or whatever's the latest
<Saviq> fginther, it would default to the newest revision in devel-proposed, could be useful to allow overriding, but not *really* necessary I'd think, for dev purposes we're all doing against the latest
<Saviq> fginther, and for anything that would require investigation, I'd say local runs are good enough
<fginther> Saviq, thanks so far... What about specifying the packages from the silo? Would it be acceptable to require a list of binary packages to install?
<Saviq> fginther, I think yes
<Saviq> fginther, we could then wrap that job with defaults for that per-project
<Saviq> fginther, or something to that effect, but yes, I think installing all generated packages would be wrong to do, dist-upgrade could be not enough... or too much, installing from distro...
<Saviq> fginther, then there's the ro requirement, phablet-test-click-setup currently only pulls from distro, it would need to understand PPAs, too
<fginther> Saviq, yep, all of those can lead to problems, but I need to make sure this isn't going to error prone
<Saviq> which brings us to https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1262879 ...
<ubot5> Ubuntu bug 1262879 in Ubuntu CI Services "There should only be one, documented, way to run tests on devices" [Undecided,Confirmed]
<Saviq> which probably needs fixing prior to the above...
* Saviq wonders if we should have autopkgtests ran on device...
<Saviq> or something standard to that end
<Saviq> anyway
<Saviq> /away
<fginther> phablet-test-click-setup pulls stuff from distro? I thought it just hit bzr branches
<Saviq> fginther, it does pull a unity8 tarball, for example
<fginther> and yes, the goal is to move to something autopkgktests like, at least in the sense that the package knows how to test itself
<fginther> Saviq, ah, unity8
<Saviq> fginther, but even if it didn't, stuf...