Raise priority of time-sensitive copy archives

[Francis J. Lacoste]

As part of their release cycle, Ubuntu does certain rebuilds for QA. The completion of these rebuilds is time-sensitive if they want to be able to act on the feedback gained through it. So the lowest priority given to those, isn't acceptable as it means that it can take weeks to complete.

We are going to give those same-priority than normal PPA build. We will also look into increasing the build farm to minimize impact on the other PPAs.

populate-archive should take a parameter to set the priority of builds coming from the copy archive as not all rebuilds are time sensitive.

[Francis J. Lacoste (flacoste)]

Matthias, can you follow-up with the different priority option you'd like on populate-archives?

[Matthias Klose (doko)]

- there should be an option to use the same priority as for the ppa's. not sure if this depends
  on the component of a package or other things

- if a rebuild is scheduled for a reason that is not time-sensitive, it would be useful to have
  a priority depending on the component (main=-10, restricted=-15, universe=-20,
  multiverse=-25).

- for both cases it would be useful to overwrite the priority of certain packages to a
  lower priority, maybe based on the package name. it doesn't make much sense to
  rebuild language packs at a high priority (language-pack*, language-support-*,
  kde-l10n-*, libreoffice-l10n, etc).

[Julian Edwards (julian-edwards)]

>We are going to give those same-priority than normal PPA build. We will also look into increasing the build farm to minimize impact on the other PPAs.

Whatever you do, if you give the same priority to rebuilds then they will *completely block* all other virtual builds until the rebuild is finished. This includes the daily builds.

I think what is needed here is a complete redesign of the scheduling algorithms.

Also I want to supersede the "copy-archive" approach that we currently have with the derived distros rebuilds as it is going to be more flexible.

[Robert Collins (lifeless)]

On Tue, Jul 5, 2011 at 7:54 PM, Julian Edwards
<email address hidden> wrote:

> I think what is needed here is a complete redesign of the scheduling
> algorithms.

I agree; I have a book here with a raft of ideas (bought by jml :)).
I'm happy to discuss some of the ramifications, or point you at the
book (but its *dense*).

-Rob

[Julian Edwards (julian-edwards)]

On Tuesday 05 July 2011 10:25:23 you wrote:
> On Tue, Jul 5, 2011 at 7:54 PM, Julian Edwards
>
> <email address hidden> wrote:
> > I think what is needed here is a complete redesign of the scheduling
> > algorithms.
>
> I agree; I have a book here with a raft of ideas (bought by jml :)).
> I'm happy to discuss some of the ramifications, or point you at the
> book (but its *dense*).

Yes. I studied queuing algorithms 20 years ago at university and remember
being quite scared at how hard it is.

[Launchpad QA Bot (lpqabot)]

Fixed in stable r13427 <http://bazaar.launchpad.net/~launchpad-pqm/launchpad/stable/revision/13427>.

[Francis J. Lacoste (flacoste)]

The fix just commited only implements the variable priority based on the component. The component was taken into account for normal builds, but not for copy builds. The same rules are now used.

Reviewing the rules though, I find that translations section packages have always a build priority of 0, even if they are a COPY build. That is obviously not correct as it means that translations section package are going to be (and were already) built first.
Bug 810716 records that aspect of the bug.

An external API script to ensure a timely processing of time sensitive build is being worked on and will complete this bug fix. Bug 810720 tracks the ability to change our queueing for better fairness.

[Robert Collins (lifeless)]

The LP changes for this are complete; the script to be written is not part of the launchpad code tree.

[Francis J. Lacoste (flacoste)]

I'll propose the script for bundling into ubuntu-archive-tools, adding a task to that project for tracking its progress.

[Francis J. Lacoste (flacoste)]

The script to handle that is now ready for review and merging in ubuntu-archive-tools. Limited testing on qastaging shows that the code is working. We'll have to test on the real build farm to get an idea of the performance improvement.

[Colin Watson (cjwatson)]

copy-build-scheduler was merged way back in November. I don't have results on its effectiveness, but we don't need to keep this bug open for that; we can always reopen this bug or open a new one if need be.