Comment 0 for bug 899972

cron.germinate currently takes on the order of ten minutes, and really doesn't have a particularly good excuse for taking so long. Fixing this would probably be enough to let us move back to 30-minute publisher cycles for Ubuntu (as we used to have with dak, back in the dawn of time), which would increase our velocity and make some of us very happy.

The main reason it's so slow is that it runs germinate as a separate process once for each of eight flavours (Ubuntu, Kubuntu, etc., each with its own seed collection) on each of five architectures. There are a few inefficiencies inherent in this approach, but the most significant is that it has to expand dependencies and build-dependencies of the seeds that are common to all the flavours eight times as often as necessary. Since the build-dependency chain in particular of the base system winds its way through a good fraction of main, this winds up being rather a lot of duplicated work.

I've been working on this problem for some time now, and have just released germinate 2.0 to support solving it properly. The most important change here is that germinate can now process multiple seed collections for a single architecture in a single instance of the core Germinator class, which allows reusing the expansions of common seeds. While I could technically have extended the command-line interface further for this, I felt that the command-line interface was already far too complicated, and decided instead to export a public, documented, and stable Python interface which can be used for this purpose.

I have been writing a Python replacement for most of cron.germinate in parallel with this to ensure that the interface is sufficient to meet Launchpad's needs, and this runs in about three minutes on my laptop (I haven't yet timed it on ftpmaster). The main remaining tasks are to write tests for it and get it reviewed.

The deployment steps should be as follows:

 * Get germinate 2.0 into the Launchpad PPA. I'm preparing backports that should be usable for this.

 * Get germinate 2.0 deployed on relevant datacentre machines. This does not require any Launchpad changes, and so should be done early; also, there was a bug fix in this release that changes the output in a few cases (I've manually verified that the changes are correct), and so it will be simpler to deploy this first so that we can more easily check that changes in the output due to the Python rewrite are harmless.

 * Change launchpad-soyuz-dependencies to depend on python-germinate. Get this deployed everywhere relevant.

 * Land my branch.