branch puller job scheduling has unfortunate behaviour in practice

[Michael Hudson-Doyle]

So this is what happens with the branch puller on staging at the moment.

In general, there are $N branches needing to be mirrored.

The puller starts up.

It asks for a branch to pull.

It gets one.

This mirrors, often very quickly, i.e. in less than 10 seconds.

When the worker exits, the scheduler is running 0 jobs, so it exits.

There are now still $N-1 branches needing to be mirrored and nothing will happen until the puller starts again.

This is most certainly not a good situation.

I can't think of any clean solution to this, but we can probably hack something up (maybe call start again on the task source as soon as we start a job?).

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michael Hudson wrote:
> Public bug reported:
>
> So this is what happens with the branch puller on staging at the moment.
>
> In general, there are $N branches needing to be mirrored.
>
> The puller starts up.
>
> It asks for a branch to pull.
>
> It gets one.
>
> This mirrors, often very quickly, i.e. in less than 10 seconds.
>
> When the worker exits, the scheduler is running 0 jobs, so it exits.
>
> There are now still $N-1 branches needing to be mirrored and nothing
> will happen until the puller starts again.
>
> This is most certainly not a good situation.
>
> I can't think of any clean solution to this, but we can probably hack
> something up (maybe call start again on the task source as soon as we
> start a job?).

I don't really understand this issue, because it doesn't sound like
job-type scheduling to me. The JobRunner iterates through all runnable
jobs, running each in turn, so I would expect a similar "scheduler" to
do the same.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkp4Y8kACgkQ0F+nu1YWqI11uACfRlLrSfhxiIOf8ZK8wwZ6gGft
e+YAn1yFbzzv4Bbg0nZNk/1s3s+389MQ
=zXTe
-----END PGP SIGNATURE-----

[Michael Hudson-Doyle (mwhudson)]

Yeah, sorry about that, I guess we mean "scheduled more like code imports".

We should probably schedule more jobs like this, btw, once we get it working :)