expire_archive_files.py doesn't expire enough cruft

[Julian Edwards]

It currently expires files for sources and binaries that were superseded or deleted. It should also expire files for obsoleted publications, at least for Ubuntu, as the files are moved to old-releases.ubuntu.com

In addition, we need to expire files for rejected PackageUploads.

(any more William?)

[William Grant (wgrant)]

Here are notes I had:

 * Find all unexpired BPRs.
 * Remove any where datecreated > (now - stay of execution)
 * Remove any from a build which is referenced by a PackageUpload that
   is not DONE or REJECTED.
 * Remove any with publications where any of these hold:
   + dateremoved is NULL
   + dateremoved > (now - stay of execution)
   + archive is private
   + archive is a PPA and blacklisted
   + archive is primary and any of these hold:
     - series is not expirable
       (expirable series are currently: warty, hoary, breezy, dapper,
       edgy, feisty, gutsy, intrepid, jaunty)
     - status is PUBLISHED or OBSOLETE (the final packages in a
       release are kept forever)
 * Expire all BPRs that remain.

What about copy archives? They are not published, so their
PackageUploads never leave ACCEPTED. We probably want a separate thing
to reject them, making them eligible for expiration?

[William Grant (wgrant)]

Note that I would be extremely averse to extending the current expire_archive_files approach: it misses stuff, and catches stuff that it shouldn't. We should rewrite it along the lines of librarian-gc.

[Julian Edwards (julian-edwards)]

On Friday 09 September 2011 00:34:56 you wrote:
> What about copy archives? They are not published, so their
> PackageUploads never leave ACCEPTED. We probably want a separate thing
> to reject them, making them eligible for expiration?

We need something separate for those. As you noted, we need a GC approach
which is running continuously for main archives and PPAs. We can't do that for
COPY archives.

[William Grant (wgrant)]

So, we can just implicitly blacklist all copy archives for now, if we want. Remember that there are still two barriers to their expiration anyway: firstly, the PackageUploads are ACCEPTED, so nothing can be expired unless and until the archive is published. And even if it is published, dateremoved IS NULL until the contents of the archive are deleted.

So we don't have to blacklist. We could just leave them under the !primary rules, and provide a script to reject all the uploads when we want them to be expired.