It is impossible to export the GMFs, even for few sites, if the number of stochastics set is large (say 10,000)
Bug #1321544 reported by
Michele Simionato
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenQuake Engine |
Fix Released
|
High
|
Michele Simionato |
Bug Description
The procedure to extract the GMF data was never fast, but in the case of 10,000 SES it may be so slow that it takes days to export even the data for few sites and for a single realization. In practice, it is unusable.
The problems are two:
1) one must perform a number of queries proportional to the number of SES (i.e. 10,000 queries for each IMT and realization)
2) the queries are slow performing array unnesting and complex joins
The solution is to read all the GMFs at once and to reorder them in Python code. It will take more memory but it can be orders of magnitude faster.
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in oq-engine: | |
status: | New → In Progress |
importance: | Undecided → High |
assignee: | nobody → Michele Simionato (michele-simionato) |
milestone: | none → 1.2.0 |
Changed in oq-engine: | |
status: | In Progress → Fix Committed |
Changed in oq-engine: | |
status: | Fix Committed → Fix Released |
To post a comment you must log in.
For a simplified Miriam's island computation the improvement in the export time has been from 30 hours (estimated) to 53 minutes. /github. com/gem/ oq-engine/ pull/1596
implemented here: https:/