Investigate parallelising cron.germinate by arch
Bug #1248867 reported by
Adam Conrad
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ubuntu-archive-publishing |
Fix Released
|
Low
|
Colin Watson |
Bug Description
It might be nice, assuming germinate isn't entirely I/O bound, to parallelise it a bit in the publisher. A quick discussion on IRC came to the conclusion that parallelising by flavour/arch combination wouldn't work so well, but parallelising purely per arch would probably work fine, and could cut the run time down a bit.
Related branches
lp:~cjwatson/launchpad/parallel-germinate
Rejected
for merging
into
lp:launchpad
- Launchpad code reviewers: Pending requested
-
Diff: 185 lines (+76/-19)2 files modifiedlib/lp/archivepublisher/scripts/generate_extra_overrides.py (+71/-10)
lib/lp/archivepublisher/tests/test_generate_extra_overrides.py (+5/-9)
lp:~cjwatson/ubuntu-archive-publishing/parallel-germinate
- William Grant: Approve (code)
- Ubuntu Package Archive Administrators: Pending requested
-
Diff: 176 lines (+74/-17)2 files modifiedlib/scripts/generate_extra_overrides.py (+70/-9)
tests/test_generate_extra_overrides.py (+4/-8)
Changed in launchpad: | |
assignee: | nobody → Colin Watson (cjwatson) |
Changed in launchpad: | |
importance: | Undecided → Low |
status: | New → Triaged |
tags: | added: soyuz-publish |
Changed in launchpad: | |
status: | Triaged → In Progress |
affects: | launchpad → ubuntu-archive-publishing |
Changed in ubuntu-archive-publishing: | |
status: | In Progress → Fix Released |
To post a comment you must log in.
We can indeed safely parallelise by architecture. Experimentation with a cloned-and-hacked version of generate- extra-overrides on snakefruit, which has very similar specs to pepo, suggests that this takes us from 162 seconds to 55 seconds (median of five runs in both cases). That's about a 3x improvement rather than the nearly 6x (for six architectures) that we'd hoped for, but still not to be sneezed at. I'll see about making the corresponding changes to generate- extra-overrides .