Split area sources into point sources (to improve distribution of calculation load)

[Damiano Monelli]

Currently, hazard calculations are parallelized over sources (i.e. a task is created for each source in a source model).

In case of source models made of 'area sources' the current distribution mechanism can produce an unevenly distribution of computational load (in a single source model, area sources can have very different sizes and therefore the number of ruptures - and the required computation time - can vary a lot). In this case a calculation can last for a long time simply because a task is assigned to a large area source. Moreover this prevent an efficient utilization of the computational infrastructure.

This problem can be reduced if an area source is indeed considered as a collection of point sources (as it is actually done in NHLIB, an area source is an extension of a point source, see: https://github.com/gem/nhlib/blob/master/nhlib/source/area.py#L24).

The 'splitting mechanism' can be be implemented as follows:

1) Given an area source, discretize the area source region (https://github.com/gem/nhlib/blob/master/nhlib/geo/polygon.py#L177)

2) Create a 'scaled' version of the original area source magnitude frequency distribution (mfd), that is take the original area source mfd and create a new evenly discretized mfd (https://github.com/gem/nhlib/blob/master/nhlib/mfd/evenly_discretized.py#L22) having occurrence rates equal to the original ones divided by the total number of points in the discretized polygon computed in 1)

3) For each point in the discretized polygon computed in 1) create a point source (https://github.com/gem/nhlib/blob/master/nhlib/source/point.py#L27), with the same attributes of the original area source but with the scaled mfd

A task can then be created for each point source.

[Lars Butler (lars-butler)]

We have discovered that it is most efficient if we group many (100-300) point sources into a single task. For simple and complex fault sources, it is still advised that only 1 source is given per task.

I think we should add a separate configuration option to define `block_size` for point sources, like so:

[hazard]
`point_source_block_size`: New. Suggested value is 200-300.
`block_size`: Already exists. No change. Suggested value is 1.

[Lars Butler (lars-butler)]

After discussing at length with the dev and hazard team, here is a summary of what we discussed:

- During the `initializing sources` phase, change dramatically the way source models are being read and stored in the database
 - Instead of simply storing each source from each source model, we instead store a copy of each source from source model for each uncertainty which applies to that source
 - The source records we store in the DB which are partitioned into calculation tasks would have a reference to the lt_realization.
 - All sources are stored in NRML object format, modified from the original source with uncertainties (if any) applied)
 - Area sources are split into point sources, which must be done after uncertainties are applied. Area sources are split by discretizing the area polygon by `area_source_discretization` and rescaling the MFD.
- This means more work to do by the control node and less work by the task workers.
 - Uncertainties are no longer applied in the task context. Instead, the only logic tree processing done by the worker is to get the GSIM logic tree branch and construct the correct GMPE.
- This change requires a fair amount of time and would cause fundamental changes in the calculation initialization and distribution code for Classical, Event-Based, and Disaggregation Hazard calculators.
- An interim solution to this is to provide a simple tool to convert an area source model in the equivalent point sources, and include this as an option in the `openquake` executable.
 - Converts a source model containing area sources, converting any area sources to a bunch of point sources given an area source discretization parameter (km).
 - Point and fault sources are not modified.
 - The command line option will be in a new category: "Pre-processing"
 - Option is "--optimize-source-model/--osm", with the arguments INPUT_FILE AREA_SRC_DISCRETIZATION OUTPUT_FILE
 - This covers a basic use case where there are no uncertainties defined in the source model logic tree. A more general solution will require a more sophisticated implementation, which should happen inside the oq-engine and be hidden from the user. These are the changes I have noted above.

Given that the "internal" solution is more complex and risky, I'm in favor of delaying this until after the v1.0.0 release of oq-engine; making this change during a period where we are trying to stabilize the engine for the 1.0 release is not a good idea.

[Lars Butler (lars-butler)]

Addition of the `point_source_block_size` parameter:

https://github.com/gem/oq-engine/pull/1153

[Lars Butler (lars-butler)]

SourceModelWriter for oq-nrmllib: https://github.com/gem/oq-nrmllib/pull/90

[Lars Butler (lars-butler)]

--optimize-source-model option for bin/openquake: https://github.com/gem/oq-engine/pull/1156