Migration script needs to be parallelized.

[Robert Bruce Park]

When the migration script was only checking published silos, it ran in about 30 seconds.

When we upgraded it to check for new commits, it ran in about 4 minutes (for ~55 silos).

Now the migration script has to iterate over every build in every package in every silo, and it takes 18 minutes to run. I should really find a way to parallelize each silo to speed things up.

[Robert Bruce Park (robru)]

Ok I think I have a reasonable parallelization plan:

Instead of having one global check-publication-migration script, there needs to be a per-silo 'status' script, which does the same thing, just considering one silo each rather than one job having a for loop over all silos.

Then all these jobs can all be set on their own timers.

Also this job should collect the same artifacts as the build job (diffs), so that those are easy to get to all in one place.

This may also require an increase in executors, as having 60 jobs fire off every 30 minutes, even if they individually only take a minute, will still overwhelm the 20 executors available. May interfere with people trying to build.

[Robert Bruce Park (robru)]

Amazingly Jenkins is smart enough to stagger all these builds so it's not like 60 builds are started simultaneously and swamping the executors. This is actually looking really good in staging.

[PS Jenkins bot (ps-jenkins)]

Fix committed into lp:cupstream2distro at revision 1227, scheduled for release in cupstream2distro, milestone Unknown