autopkgtest: allow testing against private PPAs with private results

[Andy Whitcroft]

We really would like to be able to pre-test -security updates before we copy them out to the real world.

As these sources necessarily are secret (often embargoed) it would be necessary to be able to submit them such that the nature of what is tested is not shown and the results would need to be separate and protected also. Much as we do for private builds now.

[Martin Pitt (pitti)]

This requires some changes to several components:

 - Credentials for the PPA need to be passed as part of the AMQP request, perhaps extending the syntax to "lpuser/ppaname/password". The worker would then generate setup commands to add an apt source with the necessary lpuser@password in the apt archive .
 - Requests containing a password must be disguised on running.shtml; i. e. it is ok to know that there is a test running for a private package, but not expose the test name or parameters. (Similar to what buildds do)
 - Note that this will expose the PPA credentials to operators of the infrastructure, i. e. people with ssh access to the infrastructure (me, Iain Lane, Adam Conrad, Steve Langasek, Stephane Graber)
 - The swift container for the destination PPA must not be public

It is still unclear how to expose the swift results to the right people, as the Launchpad teams are not available/exposed on Openstack credentials. Thus for each PPA we must file an RT to create an Openstack user, hand out that secret to the corresponding team, and change it whenever the team changes.

[Andy Whitcroft (apw)]

Had a discussion on the tokens to access the PPA with cjwatson. It was felt that "onetime" tokens would be possible for this and some related work is ongoing for "https git access tokens" which this might build on once that is done. Tokens could be handed out to the PPA owner, supplied with the job as in comment #1 and then revoked at the end of the job.

[Colin Watson (cjwatson)]

Doing this sensibly will require some work in Launchpad, as the existing mechanisms for dispatching private PPA builds to builders wouldn't be suitable here.

At the moment, I'm working on a general upgrade to our authorisation code to let us accept various kinds of more constrained tokens (with the constraints either living in the database, or in the token itself, or a composition of both). The initial goals are for this to be an optional addition to OAuth tokens, and to be usable as HTTPS access tokens for Launchpad git repositories. However, once all this is in place, we could do something like the following:

 * add an endpoint for private PPA authorisation to Launchpad's internal private XML-RPC server
 * work out how to integrate this with Apache so that the web server on private-ppa.launchpad.net will dynamically ask Launchpad to authenticate the user rather than using htpasswd files, which would let us fix a slew of existing bugs
 * add a mechanism for the autopkgtest system to request time-limited private PPA access tokens for PPAs its user can access
 * ensure that the autopkgtest system can revoke its own tokens once it's finished with them (although there would also be a time limit as a backstop)
 * as launchpad-buildd does, make sure that the access token does not show up in test logs, just in case revocation fails

[Colin Watson (cjwatson)]

Regarding Swift results, perhaps it would make sense to put private results in a separate container, and then proxy files through something that can check Launchpad team membership rather than linking directly to Swift.