the interval between launching a new code import job should be more flexible
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Fix Released
|
High
|
Michael Hudson-Doyle |
Bug Description
Currently, the code-import-
This is too slow: an import that doesn't require much work only takes about 10 seconds, and also launching one job every minute per machine is probably not fast enough to keep up with the current workload, never mind the future load.
It still makes sense to stagger the launching of the processes though, to reduce thrashing.
It probably makes sense to have the code-import-
for i in range(6):
job = getjobForMachine()
if job: job.launch()
time.sleep(10)
This way we get the brain dead reliability of cron, but a faster launch rate when things are working.
Related branches
- Tim Penhey (community): Approve
-
Diff: 164 lines (+63/-13)3 files modifiedcronscripts/code-import-dispatcher.py (+1/-1)
lib/lp/codehosting/codeimport/dispatcher.py (+28/-3)
lib/lp/codehosting/codeimport/tests/test_dispatcher.py (+34/-9)
Changed in launchpad-bazaar: | |
importance: | Undecided → Medium |
Changed in launchpad-bazaar: | |
status: | New → Confirmed |
tags: | added: code-import |
Changed in launchpad-code: | |
importance: | Medium → High |
assignee: | nobody → Michael Hudson (mwhudson) |
milestone: | none → 10.02 |
Changed in launchpad-code: | |
status: | Triaged → Fix Committed |
status: | Fix Committed → In Progress |
Changed in launchpad-code: | |
status: | In Progress → Fix Committed |
Changed in launchpad-code: | |
status: | Fix Committed → Fix Released |
This is actually worse than I thought. We currently cheat in the crontab to dispatch 2 imports per machine per minute. We currently have three machines, so this means we can dispatch 8640 imports per day. We currently have 1654 imports regularly being updated. Assuming that they are all updated 4 times a day (not really true, but not so far off either) we have to dispatch 6616 imports a day -- not very far away from the hard limit!