Migration script needs to be parallelized.

Bug #1517675 reported by Robert Bruce Park on 2015-11-19
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
CI Train [cu2d]
Fix Released
Undecided
Robert Bruce Park

Bug Description

When the migration script was only checking published silos, it ran in about 30 seconds.

When we upgraded it to check for new commits, it ran in about 4 minutes (for ~55 silos).

Now the migration script has to iterate over every build in every package in every silo, and it takes 18 minutes to run. I should really find a way to parallelize each silo to speed things up.

Related branches

Robert Bruce Park (robru) wrote :

Ok I think I have a reasonable parallelization plan:

Instead of having one global check-publication-migration script, there needs to be a per-silo 'status' script, which does the same thing, just considering one silo each rather than one job having a for loop over all silos.

Then all these jobs can all be set on their own timers.

Also this job should collect the same artifacts as the build job (diffs), so that those are easy to get to all in one place.

This may also require an increase in executors, as having 60 jobs fire off every 30 minutes, even if they individually only take a minute, will still overwhelm the 20 executors available. May interfere with people trying to build.

Robert Bruce Park (robru) wrote :

Amazingly Jenkins is smart enough to stagger all these builds so it's not like 60 builds are started simultaneously and swamping the executors. This is actually looking really good in staging.

PS Jenkins bot (ps-jenkins) wrote :

Fix committed into lp:cupstream2distro at revision 1227, scheduled for release in cupstream2distro, milestone Unknown

Changed in cupstream2distro:
status: Triaged → Fix Committed
Changed in cupstream2distro:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers