I had further follow-up with Saviq on #ubuntu-ci-eng.

In summary:
 * A job that runs everything serially would work as a first step (better then nothing)
 * Installed image should be configurable (to a channel or number) and default to devel-proposed
 * Specifying the set of binaries to install from the silo is ok (believed to be a better option then install all of them or doing a dist-upgrade).

<fginther> Saviq, is a job that runs all the ap suites in serial of any use?
<Saviq> fginther, sure
<Saviq> fginther, would be a stop-gap, but still faster than running them manually
<Saviq> fginther, and, fwiw, more reliable
<fginther> Saviq, what would the image baseline need to be? proposed?
<Saviq> fginther, that'd probably be used most often
<Saviq> fginther, but since it's flashing every time anyway, would that really matter?
<Saviq> fginther, should be possible to flash with any channel / revision even
<fginther> Saviq, I guess I meant proposed vs stable, etc.
<Saviq> fginther, in terms of system image, not distro, right?
<fginther> Saviq, yes, I'm referreing to the system image, my terminology may be off
<Saviq> fginther, but yeah, same applies, since the devices are being flashed every time anyway, it should be possible to select which channel / revision to flash
<Saviq> fginther, channel = devel-proposed, revision = 283 or whatever's the latest
<Saviq> fginther, it would default to the newest revision in devel-proposed, could be useful to allow overriding, but not *really* necessary I'd think, for dev purposes we're all doing against the latest
<Saviq> fginther, and for anything that would require investigation, I'd say local runs are good enough
<fginther> Saviq, thanks so far... What about specifying the packages from the silo? Would it be acceptable to require a list of binary packages to install?
<Saviq> fginther, I think yes
<Saviq> fginther, we could then wrap that job with defaults for that per-project
<Saviq> fginther, or something to that effect, but yes, I think installing all generated packages would be wrong to do, dist-upgrade could be not enough... or too much, installing from distro...
<Saviq> fginther, then there's the ro requirement, phablet-test-click-setup currently only pulls from distro, it would need to understand PPAs, too
<fginther> Saviq, yep, all of those can lead to problems, but I need to make sure this isn't going to error prone
<Saviq> which brings us to https://bugs.launchpad.net/ubuntu-ci-services-itself/+bug/1262879 ...
<ubot5> Ubuntu bug 1262879 in Ubuntu CI Services "There should only be one, documented, way to run tests on devices" [Undecided,Confirmed]
<Saviq> which probably needs fixing prior to the above...
* Saviq wonders if we should have autopkgtests ran on device...
<Saviq> or something standard to that end
<Saviq> anyway
<Saviq> /away
<fginther> phablet-test-click-setup pulls stuff from distro? I thought it just hit bzr branches
<Saviq> fginther, it does pull a unity8 tarball, for example
<fginther> and yes, the goal is to move to something autopkgktests like, at least in the sense that the package knows how to test itself
<fginther> Saviq, ah, unity8
<Saviq> fginther, but even if it didn't, stuff that's in silos isn't there in trunks anyway...
<fginther> Saviq, how do you test click packages (or do you)
<fginther> ?
<Saviq> fginther, phablet-test-click-setup + phablet-test-run
<Saviq> fginther, but when I add a unity8 silo, p-t-c-s fails, 'cause it's trying to get a tarball for the installed unity8 version from distro
<fginther> Saviq, sorry, I meant if a click package needs to be built, those are being built by silos
<fginther> Saviq, I've seen that problem
<Saviq> fginther, I'm not sure, but if they are indeed built by silos (I'm not sure they are, at least not for uploading to the store), p-t-c-s would need to know how to deal with that, too, and either branch from the silo (it's currently possible from http://people.canonical.com/~didrocks/citrain/silos/) or grab tarballs from the PPAs
<fginther> Saviq, we're trying to solve the same issue with the ci-airline, I was just curious if the ci-train how a known way for dealing with click packages