Improve host load balancing on ImportD service

[Stefan Stasik]

Hello: Currently we have 4 ImportD job servers that process code imports on Launchpad. We use a basic crontab system to launch the processing scripts on a staggered basis (using shell 'sleep') to try to balance how the 4 hosts process the incoming jobs. We need to come up with a improved way of handling these jobs so that the load-balancing happens in a more intelligent way. One of the servers is much more powerful than the others, so a round-robin approach doesn't work well.

Thank you, Stefan Stasik

[Tom Haddon (mthaddon)]

Does the --max-jobs option do this?

[Tom Haddon (mthaddon)]

It seems this isn't working as expected. See --max-jobs=3 while galapagos is clearly running 4 jobs.

[Michael Hudson-Doyle (mwhudson)]

On 27/03/10 01:31, Tom Haddon wrote:
> It seems this isn't working as expected. See --max-jobs=3 while
> galapagos is clearly running 4 jobs.
>
> ** Attachment added: "Screenshot.png"
> http://launchpadlibrarian.net/42049427/Screenshot.png

The second invocation in the cron job still says 4.

I guess we should really kill the need for that hack.

Cheers,
mwh

[Tom Haddon (mthaddon)]

Ah, thx - now fixed to use --max-jobs=3 on both runs.

[Steve McInerney (spm)]

we had a funky yesterday, where a single import (afaict - and was the linux kernel) caused sufficient load on galapagos that later importds getting fired off would be sufficiently slowed down that we ended up having a bunch of new importds scheduled from cron all gradually working their way to failing to get a lock and die.

Which also explained the sudden load spike.
https://pastebin.canonical.com/29841/

It did sort itself, but perhaps a light weight wrapper to trap this behaviour may be worthwhile?

[Tim Penhey (thumper)]

Right now this is a low priority. The high priority bit is to make the git imports much faster. Jelmer has already done much of this and we are testing on staging. We are talking orders of magnitude faster, so that'd be good.