Require a way to copy [P]PPA packages into Ubuntu

[Julian Edwards]

We want to be able to copy PPA or Private PPA packages into Ubuntu, optionally via the distroseries upload queues as necessary.

This is required to be able to:
 * Make an API to unembargo security packages, so we can get rid of the shell script that currently does it
 * Copy partner packages from PPAs (when that's allowed) into Ubuntu (Bug #244159)
 * Allow anyone who has upload access to Ubuntu to be able to go to a PPA and initiate a package copy to Ubuntu

This will be modelled as a package upload but will not re-process the upload itself; the package will just pass through the queue and follow the existing queue rules. Security (unembargoed) uploads should not stay in the queue.

[Julian Edwards (julian-edwards)]

This is dependent on the work being done in https://blueprints.edge.launchpad.net/soyuz/+spec/soyuz-delayed-copies which is nearly complete.

[Celso Providelo (cprov)]

archive.syncSource API is already able to copy sources (binaries and custom uploads) from private PPA directly to the ubuntu main archives (PRIMARY and PARTNER).

This feature is *only* available to members of the ubuntu-drivers and ubuntu-security teams atm.

Here is an example:

{{{
    ubuntu = lp.distributions['ubuntu']
    primary, partner = ubuntu.archives
    security_ppa = lp.people['ubuntu-security'].archive
    primary.syncSource(
        source_name='hello', version='2.2-3ubuntu4',
        from_archive=security_ppa, include_binaries=True,
        to_series='jaunty', to_pocket='Security')
}}}

Version conflicts and other copy mistakes will result in failures and consequently no changes.

On success (nothing is returned), It results in an ACCEPTED item in the corresponding upload queue (ubuntu/jaunty/+queue) containing the source in question, its successfully built binaries and custom uploads available in the original archive (partially built sources will result in new builds in the destination archive).

The accepted 'delayed-copies' will be processed during the hourly publication cycle, bugs mentioned in the changelog will be closed and private files will become public at this point. The queue item will be marked as DONE.

No email notification will be generated (missing feature, sorry).

Next step is extend the permissions to execute copies to all ubuntu uploaders, respecting their permissions.

[Julian Edwards (julian-edwards)]

This is related to bug 244163 and can't be fixed yet for the same reasons.

[Robert Collins (lifeless)]

Don't we need to force rebuilds in the primary archive always except for security unembargos? Doing otherwise will trivially cause library mismatches, FTBFS scenarios and so on.

[Julian Edwards (julian-edwards)]

Yes. This can be rolled up into the Package Copy Job stuff quite trivially now, since that is already in use for syncs from Debian. We just need the extra UI selection and view code logic.

[Launchpad QA Bot (lpqabot)]

r15480 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15480) is part of this bug's fix.

[Colin Watson (cjwatson)]

This OOPSes when processing the PCJ:

Traceback (most recent call last):
  Module lazr.jobrunner.jobrunner, line 194, in runJobHandleError
    self.runJob(job, fallback)
  Module lp.services.job.runner, line 270, in runJob
    super(BaseJobRunner, self).runJob(IRunnableJob(job), fallback)
  Module lazr.jobrunner.jobrunner, line 162, in runJob
    job.run()
  Module lp.soyuz.model.packagecopyjob, line 495, in run
    self.attemptCopy()
  Module lp.soyuz.model.packagecopyjob, line 562, in attemptCopy
    unembargo=self.unembargo)
  Module lp.soyuz.scripts.packagecopier, line 686, in do_copy
    previous_version=old_version)
  Module lp.soyuz.adapters.notification, line 193, in notify
    bprs=bprs)
  Module lp.soyuz.adapters.notification, line 477, in get_upload_notification_recipients
    info = fetch_information(spr, bprs, changes)
  Module lp.soyuz.adapters.notification, line 631, in fetch_information
    changesfile = spr.aggregate_changelog(previous_version)
  Module lp.soyuz.model.sourcepackagerelease, line 649, in aggregate_changelog
    for block in Changelog(changelog.read()):
  Module lp.services.librarian.model, line 171, in read
    self.open(timeout=timeout)
  Module lp.services.librarian.model, line 161, in open
    self._datafile = self.client.getFileByAlias(self.id, timeout)
  Module lp.services.librarian.client, line 476, in getFileByAlias
    return self._connect_read(url, try_until, aliasID)
  Module lp.services.librarian.client, line 491, in _connect_read
    raise LookupError(aliasID)
LookupError: 103960041

I'll need to figure out what's going on here; I assume it's either a missing transaction.commit or passing the wrong (private) objects to notify.

Archive.syncSource (the delayed-copy path) still works. So we can *probably* deploy this anyway, but the OOPS needs to be fixed ASAP.

[Colin Watson (cjwatson)]

Also, non-unembargoing Archive.copyPackage calls are unaffected.

[Colin Watson (cjwatson)]

And unembargo-package.py is unaffected, since it doesn't pass send_email=True; it's only PCJs that do that. So, in summary, the only broken thing is the new API which must be explicitly selected, making this safe to deploy despite the problems.

[Launchpad QA Bot (lpqabot)]

r15486 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15486) is part of this bug's fix.

[Colin Watson (cjwatson)]

r15486 seems to have cleared this up. The only mild oddity is that the BinaryPackageBuild is still on a URL for the private archive (https://dogfood.launchpad.net/~cjwatson/+archive/dogfood-private/+build/3459155), but it's actually public; this appears to be normal. Publication went off without a hitch as far as I can tell.

[Micah Gersten (micahg)]

Yes, those URLS are normal:
https://launchpad.net/ubuntu/+source/nova/2011.3-0ubuntu6.8
https://launchpad.net/~ubuntu-security/+archive/ppa/+build/3568729

[Colin Watson (cjwatson)]

Apart from removing the old copy-package.py script, I think the only thing that's missing here is a way to copy packages into the Ubuntu primary archive using the UI. Of course, as mentioned, this requires care since we should only allow including binaries in certain circumstances. This isn't urgent in any case since the API is now sufficient for us.

[Launchpad QA Bot (lpqabot)]

r15772 in stable (http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/15772) is part of this bug's fix.