Compute ground motion values on the fly for risk event based calculations

[matley]

We need to support the calculation of ground motion values on the fly to be used in a risk event based calculation.

Given the id <foo> of an output of type "ses" computed by a hazard event based calculation it should be possible to run:

./bin/openquake --rr job_risk.ini --ho <foo>

Implementation proposal:

We enhance the GroundMotionValuesGetter class in the hazard_getters module in order to allow it to compute on the fly the needed ground motion values.

As the ground motion values are values deriving from a stochastic process we need to ensure that the correlation between such numbers are not lost.

Roughly speaking, as in a Risk Event Based calculator the computation is split across assets (i.e. sites) we need to ensure that all the random numbers generated in a task (e.g. a total stddev residual for a particular rupture) is computed properly (e.g. the same rupture gets the same residual in different tasks). Similar attention should be taken for managing the generation of intra-event, inter-event residuals (as for the former spatial correlation has to be considered).

The proposed solution involves the generation upfront of a number of seed values used to initialize the random number generator in the different tasks. Then, we use a simplified version of `openquake.hazardlib.calc.ground_motion_values` that accepts the random numbers input to compute the ground motion values.

As to compute ground motion values, the full ProbabilisticRupture object is needed, we also need to change the schema of the database in order to save the whole rupture (as a pickled object).

Related issue: https://bugs.launchpad.net/oq-hazardlib/+bug/1204591

[matley (matley)]

https://github.com/gem/oq-engine/pull/1234/

[matley (matley)]

The first runs on the system `HOPE` (1 machine, 20 workers) with the configuration "Miriam's Virtual Island" are promising.

With the standard approach, the hazard part takes 41' 39'', while the risk one 5' 16''. The latter consumes at most 944Mb of memory per worker. With the new approach the hazard takes 34' 01'' (as we are not computing and saving gmvs), the risk 20' 43'' but consuming only 825Mb. As the new approach does not cost in terms of communication between the database on the controller node and the workers this is a good gain.

I am going to run the same analysis on the cluster

[matley (matley)]

More experiments on the system `COBAIN` (1 machine, 64 core).

Hazard took 20', Risk 15'. Memory consumption per task was at most 754Mb.

[matley (matley)]

Detailed statistics (job_id with 1 and 2 are the hazard and the risk ones, respectively):

openquake=> select oq_job_id, operation, duration, tot_duration, pymemory from uiapi.performance_view ;
 oq_job_id | operation | duration | tot_duration | pymemory
-----------+----------------------------+----------------------+-------------------+----------------------------
         1 | computing ses | 54.5785155575307 | 54578.5155575307 | 1.8671875000000000
         1 | initialize_site_model | 1.61293888092 | 1.61293888092 | 3.3750000000000000
         1 | initialize_sources | 0.123151063919 | 0.123151063919 | 3.8398437500000000
         1 | loading calculation object | 0.00821465206146259 | 8.21465206146259 | 0.14453125000000000000
         1 | parse_risk_models | 0.0169970989227 | 0.0169970989227 | 0.15234375000000000000
         1 | reading site collection | 0.0495810952186581 | 49.5810952186581 | 4.8320312500000000
         1 | saving ses | 1.19606592893602 | 1196.06592893602 | 0.00390625000000000000
         1 | storing task id | 0.000456260442733764 | 0.456260442733764 | 0.05078125000000000000
         1 | totals per task | 55.9221015367495 | 55922.1015367495 | 14.3906250000000000
         2 | associating assets->site | 0.885415977047247 | 54.8957905769293 | 0.01171875000000000000
         2 | computing gmvs | 589.520921441774 | 36550.29712939 | 555.2148437500000000
         2 | computing individual risk | 81.5996769128242 | 5059.1799685951 | 363.6484375000000000
         2 | computing risk statistics | 0.00400700492243661 | 0.24843430519107 | 0.000000000000000000000000
         2 | loading calculation object | 0.0093039197306479 | 0.57684302330017 | 0.000000000000000000000000
         2 | post processing | 105.123908043 | 105.123908043 | 1699.9882812500000000
         2 | saving individual risk | 0.906906227911677 | 56.228186130524 | 0.000000000000000000000000
         2 | saving risk statistics | 0.00177096166918386 | 0.109799623489399 | 0.000000000000000000000000
         2 | storing task id | 0.000427592185235758 | 0.026510715484617 | 0.000000000000000000000000
         2 | totals per task | 687.754646135984 | 42640.788060431 | 754.0000000000000000