Handle supervision in oq-engine-server

[Lars Butler]

In order to run oq-engine calculations in oq-engine-server, we have to disable the supervisor forking (see https://github.com/gem/oq-engine/blob/584a5265277ce27ae90b362e3bc036f64998470c/openquake/engine/engine.py#L393), otherwise we leak celeryd processes.

We can avoid this leaking issue by not forking, but we lose some important monitoring done by the supervisor process. Namely, work node monitoring (if a worker crashes/goes offline, we abort the calculation).

oq-engine-server will need to implement some other kind of monitoring, perhaps something involving celerymon.

[Michele Simionato (michele-simionato)]

https://github.com/gem/oq-engine/pull/1321

[Lars Butler (lars-butler)]

Hi Michele,

It looks like you've done some great cleanup here!

Is there a corresponding patch to the oq-engine-server? Just removing supervision from the oq-engine doesn't really solve this problem, does it? And if you remove all of this, how do you detect failed celery nodes? Is this feature still there somewhere? (It's a little unclear from the pull request, though I haven't had a chance to test it.)

[Michele Simionato (michele-simionato)]

Hi Lars, actually there is a task scheduled this sprint for the error management in the engine server. The detection of failed nodes is still there, see https://github.com/gem/oq-engine/blob/master/openquake/engine/engine.py#L488