Simplify the task distribution in the risk calculators

Bug #1376157 reported by Michele Simionato
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenQuake Engine
Fix Released
High
Michele Simionato

Bug Description

At the moment we have a complex mechanism: the controller node spawns a few supertasks (one for each taxonomy) and the supertasks spawn several subtasks (one for each block of 100 assets). The problem, apart for complexity and for being a special case, different from the mechanism used in the hazard calculators, is that the number of assets is hard-coded to 100. If you have an exposure with say 10^6 assets, 10^4 tasks will be generated: too many. In general we know that the number of tasks generated must be limited, otherwise rabbitmq will die. Even when rabbitmq is able to cope with many tasks, we know that the performance is best when the number of tasks is not much bigger than the number of cores.

The proposal is:

1) split the distribution in two phases: the first phase prepares the risk input, the second phase performs the actual computation
2) honor the concurrent_tasks parameter, as in the hazard calculators.

Revision history for this message
Michele Simionato (michele-simionato) wrote :
Changed in oq-engine:
importance: Undecided → High
assignee: nobody → Michele Simionato (michele-simionato)
milestone: none → 1.0.1
status: New → In Progress
Changed in oq-engine:
status: In Progress → Fix Committed
Changed in oq-engine:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.