Ampoule job runner should not run in launchpad environment

[Aaron Bentley]

The ampoule-based job runner is slower than the synchronous job runner because it needs to start up two launchpad environments-- one for the master process and one for the subprocess.

Instead, the ampoule-based job runner should defer all work that requires a launchpad environment to the subprocess.

Currently, two significant operations require a launchpad environment:
1. Finding available jobs to run
2. Reporting oopses when a job terminates uncleanly

Since our ampoule-based job running script may become a daemon, the master process should be as simple as possible. By shoving all this work into subprocesses, we reduce the scope for resource leaks and abnormal termination in the master process. Child processes can, of course, be killed.

[Jonathan Lange (jml)]

By "start up two launchpad environments", I presume you mean "process launchpad's zcml"?

Why not have all the Launchpad stuff done in the job runner, leaving the subprocess nice and fast to start?

[Aaron Bentley (abentley)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jonathan Lange wrote:
> By "start up two launchpad environments", I presume you mean "process
> launchpad's zcml"?

That is all we currently do, but there may be more that we should be doing.

> Why not have all the Launchpad stuff done in the job runner, leaving the
> subprocess nice and fast to start?

That would mean that the subprocess would not have access to any of our
normal facilities, such as zope utilities or the database. Operations
which required those facilities would have to be performed by the master
process. We want to make the master process simpler, but this would
make it more complex instead. It would greatly increase the likelihood
of resource leaks in the master process. Jobs themselves would become
harder to write, and I expect some would have to be split into several
AMP commands.

Aaron
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkspO4IACgkQ0F+nu1YWqI19PACeO94L7WcQ/7UEaAXquyw/hKsG
OkAAn0Grl0cSu02DG7tuxyRZqgb7gu4L
=mGM8
-----END PGP SIGNATURE-----

[Jonathan Lange (jml)]

On Thu, Dec 17, 2009 at 6:56 AM, Aaron Bentley <email address hidden> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jonathan Lange wrote:
>> By "start up two launchpad environments", I presume you mean "process
>> launchpad's zcml"?
>
> That is all we currently do, but there may be more that we should be
> doing.
>
>> Why not have all the Launchpad stuff done in the job runner, leaving the
>> subprocess nice and fast to start?
>
> That would mean that the subprocess would not have access to any of our
> normal facilities, such as zope utilities or the database.  Operations
> which required those facilities would have to be performed by the master
> process.  We want to make the master process simpler, but this would
> make it more complex instead.  It would greatly increase the likelihood
> of resource leaks in the master process.  Jobs themselves would become
> harder to write, and I expect some would have to be split into several
> AMP commands.
>

OK. It's a matter of trade-offs then. The branch puller forces almost
all of the database stuff onto the master process precisely to avoid
the long start-up times of processing zcml. When you have tens of
thousands of jobs a day, seconds count.

I agree that it does make resource leaks more likely. I can believe
that it would make jobs harder to write, but I'd like to know more of
the specifics.

I don't want to block progress on this bug by demanding perfection
rather than improvement, however, perhaps it's worth challenging Gary
& the Foundations team with the slow ZCML problem. After all, if it
were lightning fast, none of this would be a problem (and we could
simplify much of our existing code).

jml

[Aaron Bentley (abentley)]

I don't think the start-up costs of subprocesses are relevant here, because we will not be starting new subprocesses for each job. Ampoule is a process pool, and we can reuse the subprocesses as many times as we like.

[Jonathan Lange (jml)]

On Thu, Dec 17, 2009 at 5:22 PM, Aaron Bentley <email address hidden> wrote:
> I don't think the start-up costs of subprocesses are relevant here,
> because we will not be starting new subprocesses for each job.  Ampoule
> is a process pool, and we can reuse the subprocesses as many times as we
> like.

Ahh, I see. That makes much more sense.

jml

[Robert Collins (lifeless)]

FWIW oopses can be done outside zope/zcml easily now.