Simplify the task distribution in the risk calculators
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenQuake Engine |
Fix Released
|
High
|
Michele Simionato |
Bug Description
At the moment we have a complex mechanism: the controller node spawns a few supertasks (one for each taxonomy) and the supertasks spawn several subtasks (one for each block of 100 assets). The problem, apart for complexity and for being a special case, different from the mechanism used in the hazard calculators, is that the number of assets is hard-coded to 100. If you have an exposure with say 10^6 assets, 10^4 tasks will be generated: too many. In general we know that the number of tasks generated must be limited, otherwise rabbitmq will die. Even when rabbitmq is able to cope with many tasks, we know that the performance is best when the number of tasks is not much bigger than the number of cores.
The proposal is:
1) split the distribution in two phases: the first phase prepares the risk input, the second phase performs the actual computation
2) honor the concurrent_tasks parameter, as in the hazard calculators.
Changed in oq-engine: | |
status: | In Progress → Fix Committed |
Changed in oq-engine: | |
status: | Fix Committed → Fix Released |
https:/ /github. com/gem/ oq-engine/ pull/1542 /github. com/gem/ oq-commonlib/ pull/43 /github. com/gem/ oq-risklib/ pull/77
https:/
https:/