the interval between launching a new code import job should be more flexible

Bug #236973 reported by Michael Hudson-Doyle
4
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Michael Hudson-Doyle

Bug Description

Currently, the code-import-dispatcher script just launches a single job each time it is run, and it is run once a minute from cron.

This is too slow: an import that doesn't require much work only takes about 10 seconds, and also launching one job every minute per machine is probably not fast enough to keep up with the current workload, never mind the future load.

It still makes sense to stagger the launching of the processes though, to reduce thrashing.

It probably makes sense to have the code-import-dispatcher script launch to do effectively this:

for i in range(6):
    job = getjobForMachine()
    if job: job.launch()
    time.sleep(10)

This way we get the brain dead reliability of cron, but a faster launch rate when things are working.

Related branches

Changed in launchpad-bazaar:
importance: Undecided → Medium
Changed in launchpad-bazaar:
status: New → Confirmed
Revision history for this message
Michael Hudson-Doyle (mwhudson) wrote :

This is actually worse than I thought. We currently cheat in the crontab to dispatch 2 imports per machine per minute. We currently have three machines, so this means we can dispatch 8640 imports per day. We currently have 1654 imports regularly being updated. Assuming that they are all updated 4 times a day (not really true, but not so far off either) we have to dispatch 6616 imports a day -- not very far away from the hard limit!

Jonathan Lange (jml)
tags: added: code-import
Changed in launchpad-code:
importance: Medium → High
assignee: nobody → Michael Hudson (mwhudson)
milestone: none → 10.02
Changed in launchpad-code:
status: Triaged → Fix Committed
status: Fix Committed → In Progress
Changed in launchpad-code:
status: In Progress → Fix Committed
Changed in launchpad-code:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.