Experiment with the idea of removing celery from the engine server

[Michele Simionato]

It is probably possible to remove the architectural complications and limitations due to the choice of using celery as task queue manager in the engine-server. Perhaps using a simple concurrent.futures.ThreadPoolExecutor is enough for our use case. In the future we may also want to use a third party priority queue software. The bug here suggests an investigation on alternatives to celery for the engine-server.

[Michele Simionato (michele-simionato)]

This is of relevance for the engine lite and also for hazardlib and risklib, that could grow utilities to run code in parallel.

[Michele Simionato (michele-simionato)]

Here is a list of issues we are having with celery.

1. PYTHONPATH issues. If you are not very careful, the wrong celeryconfig.py will be taken, and you will start twice the celery of the engine. All of us got tricked by those issues, several times, even knowing that they were there. It is more tricky to get it right than one would expect.

2. There are two servers to start/restart and two logs to watch. It is very easy to get confused, to restart the wrong celery etc

3. celery tasks have limitations, for instance you cannot spawn a subprocess from a celery task. And we wanted to spawn a psql subprocess to copy the output from the engine database to the icebox database.

On top of that, it is simply easier to reason about if the job queue is totally separated from the tasks queue and actually uses a completely different technology. concurrent.futures is in the standard library, it is easy to understand and it works perfectly fine for the use case of the engine-server, i.e. to spawn jobs on the controller machine, so it is a natural choice.

[Michele Simionato (michele-simionato)]

celery was removed from the engine-server here: https://github.com/gem/oq-engine-server/pull/21