Add support for beta distributions

[Vitor Silva]

Currently the uncertainty in the loss ratios can only be modelled with a lognormal distribution. This uncertainty can also be modelled with a beta distribution, which can also be defined by two parameters: a mean and a standard deviation or an alpha and beta values.
This feature affects the loss ratio sampler and the creation of the loss ratio exceedance matrices.

Openquake now should not assume the distribution of loss ratio as lognormal, but to read in the vulnerability file which type of distribution is being considered.

----------
Open Issues:

- Which calculators does this change affect?
- Are there any changes to output formats?
- What are the changes to input files?
  - If there are changes, we need a set of demo input files which can exercise this functionality. We also need demo files to write a QA test.
- We can probably utilize scipy.stats.beta [1] for this bug. Where do the alpha and beta values come from?

[1] - http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html

[Lars Butler (lars-butler)]

More details needed.
- Which calculators does this change affect?
- Are there any changes to output formats?
- Is there test data available (input + expected results)?
- Are the formulae for this new type of calculation somewhere? If so, where?

[Vitor Silva (vitor-silva)]

Since this changes the way the uncertainty in the loss ratios is handled, it will affect all the calculators that make use of vulnerability models. Furthermore, instead of having the attribute probabilisticDistribution within the vulnerability model always equal to "LN" (lognormal), now this attribute can also be equal to "BT" (beta).

In the Scenario Risk Calculator and Probabilistic Event-based Risk Calculator it affects the way the loss ratio is sampled. No data can be provided since the values are randomly sampled.

In the Classical PSHA-based Risk Calculator it affects the way the probabilities of exceedance are computed in the loss ratio exceedance matrix. Test data has been attached in which a vulnerability function (whose uncertainty is modelled using a beta distribution) is used to generate a Loss ratio Exceedance Matrix. Note that frequently alpha and beta parameters are used to characterize this function instead of the regular mean and standard deviation. The required formulae to compute alpha and beta have also been added to the document.

To random sample beta distributions or to compute its cumulative value, the ScyPy library needs to be used, as most of these functions do not have an analytical expression.

[Lars Butler (lars-butler)]

It appears we'll need to add 'BT' to the distribution type enum in the nrml spec: https://github.com/gem/openquake/blob/master/openquake/nrml/schema/0.3/nrml_common.xsd#L64

I see where the change of distribution type affects the Classical Risk calculator: https://github.com/gem/openquake/blob/master/openquake/calculators/risk/classical/core.py#L113

However, I don't see how this will affect the Scenario and Event-Based calculators. Also, I don't see any random sampling going on in the Scenario or EB calcs; if that were the case, we can provide seeds to random algorithms to produce consistent and predictable results.

[Lars Butler (lars-butler)]

Where do the alpha and beta values come from?

Also, is there a prototype how the beta distribution scipy functionality is utilized. This is not a part of the scipy lib I'm familiar with so an example would be very helpful.

[Lars Butler (lars-butler)]

Are there any changes to input files? Is there a set of demo files which exercises this functionality?

[Vitor Silva (vitor-silva)]

As mentioned above, alpha and beta are computed using the formulae described in the attached file.

I'm not familiar with scipy lib either. Some examples can be found here: http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html

Some results to verify this function have been provided already.

There are no changes in the input files format. This feature can be tested as a unit test and compiling input files to carry out an end to end test is not necessary.

[beatpanic (kpanic)]

pull request: https://github.com/gem/openquake/pull/671

[beatpanic (kpanic)]

Under review, near to hit master

[Lars Butler (lars-butler)]

Per a conversation with beatpanic, the formula in the image in the attached spreadsheet (https://bugs.launchpad.net/openquake/+bug/901955/comments/2) labels a variable as 'Variance'; it should be corrected to 'Std Dev'.

[beatpanic (kpanic)]

The formulae is correct (after clarifying with Vitor) it's just that sigma is standard deviation and sigma square is the variance

[Vitor Silva (vitor-silva)]

The formula in the attached file is correct.
It uses a sigma symbol, which stands for the standard deviation and not the variance.