The conclusion here is that in order to maximize utilization of a compute cluster, the following steps should be taken:
- In celeryconfig.py, set CELERY_ACKS_LATE=True, CELERYD_PREFETCH_MULTIPLIER=1. This ensures that each worker will only take one task at a time and will greedily prefetch. This is the recommend configuration for long-running tasks, as advised by asksol, the Celery author. See also: http://docs.celeryproject.org/en/master/userguide/optimizing.html#prefetch-limits
- In the openquake.cfg, the recommended number of `concurrent_tasks` is 2*the number of worker processes.
These changes ensure optimal task distribution, which most important for large calculations.
The conclusion here is that in order to maximize utilization of a compute cluster, the following steps should be taken:
- In celeryconfig.py, set CELERY_ ACKS_LATE= True, CELERYD_ PREFETCH_ MULTIPLIER= 1. This ensures that each worker will only take one task at a time and will greedily prefetch. This is the recommend configuration for long-running tasks, as advised by asksol, the Celery author. See also: http:// docs.celeryproj ect.org/ en/master/ userguide/ optimizing. html#prefetch- limits
- In the openquake.cfg, the recommended number of `concurrent_tasks` is 2*the number of worker processes.
These changes ensure optimal task distribution, which most important for large calculations.
Related code changes: https:/ /github. com/gem/ oq-engine/ pull/984