Comment 2 for bug 1092050

Revision history for this message
Lars Butler (lars-butler) wrote :

The conclusion here is that in order to maximize utilization of a compute cluster, the following steps should be taken:

- In celeryconfig.py, set CELERY_ACKS_LATE=True, CELERYD_PREFETCH_MULTIPLIER=1. This ensures that each worker will only take one task at a time and will greedily prefetch. This is the recommend configuration for long-running tasks, as advised by asksol, the Celery author. See also: http://docs.celeryproject.org/en/master/userguide/optimizing.html#prefetch-limits
- In the openquake.cfg, the recommended number of `concurrent_tasks` is 2*the number of worker processes.

These changes ensure optimal task distribution, which most important for large calculations.

Related code changes: https://github.com/gem/oq-engine/pull/984