It is impossible to export the GMFs, even for few sites, if the number of stochastics set is large (say 10,000)

Bug #1321544 reported by Michele Simionato
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenQuake Engine
Fix Released
High
Michele Simionato

Bug Description

The procedure to extract the GMF data was never fast, but in the case of 10,000 SES it may be so slow that it takes days to export even the data for few sites and for a single realization. In practice, it is unusable.
The problems are two:

1) one must perform a number of queries proportional to the number of SES (i.e. 10,000 queries for each IMT and realization)
2) the queries are slow performing array unnesting and complex joins

The solution is to read all the GMFs at once and to reorder them in Python code. It will take more memory but it can be orders of magnitude faster.

description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
description: updated
Changed in oq-engine:
status: New → In Progress
importance: Undecided → High
assignee: nobody → Michele Simionato (michele-simionato)
milestone: none → 1.2.0
Revision history for this message
Michele Simionato (michele-simionato) wrote :

For a simplified Miriam's island computation the improvement in the export time has been from 30 hours (estimated) to 53 minutes.
implemented here: https://github.com/gem/oq-engine/pull/1596

description: updated
Changed in oq-engine:
status: In Progress → Fix Committed
Changed in oq-engine:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.