Several demos are still lacking QA tests

[Lars Butler]

All of the demos in OpenQuake should have QA tests. Recently, a couple of bug fixes revealed additional bugs and caused two of our QA tests to break, but we didn't know about for several days. (See bug # 901092 and bug # 901091.)

At the time this bug was written, the following demos are lacking in QA test coverage:

- area_source_demo_hazard
- classical_psha_based_risk
- probabilistic_event_based_risk
- scenario_risk
- simple_fault_demo_hazard

[Lars Butler (lars-butler)]

Vitor Silva has provided some files for writing QA tests for the Classical and Classical RBCR Risk calculators.

[Paul Henshaw (paul-sl-henshaw)]

Created bug #906911 for minimal smoke tests as stop gap solution

[Paul Henshaw (paul-sl-henshaw)]

Sorry, comment #2 should have been
Created bug #943890 for minimal smoke tests as stop gap solution

[Paul Henshaw (paul-sl-henshaw)]

Created bug #947149 to review risk tests.

[Paul Henshaw (paul-sl-henshaw)]

Created bug #947155 to review hazard tests

[Andrea Cerisara (acerisara)]

This is the status of the QA tests related to the risk calculators.

BCR calculator: QA test present for the classical mode. The test for the event based mode is still missing.

Classical PSHA: QA test present, but incomplete. It checks the loss curves are correctly computed, but taking data from the database and not from the final nrml file produced. Also, it doesn't check the loss ratio curves and the conditional maps.

Scenario: no QA tests present, but there's in the suite a test that runs demos/scenario_risk and verifies it completes successfully.

Probabilistic event based: no QA tests present, and the demo we have in the repo (demos/probabilistic_event_based_risk) takes a lot of time to complete (approximately 30mins on my machine).

[Lars Butler (lars-butler)]

What about calculators with --output-type=xml? We've recently had problems with things breaking because there was no QA coverage for calculators run with this output mode.

We should make sure we cover these cases as well.

[Lars Butler (lars-butler)]

Related to bug # 947155.

Hazard QA Coverage:

classical:
  - end-to-end? Yes
  - DB? Yes
  - XML? Yes
  - demo suitable for QA test? Yes

event-based:
  - end-to-end? No
  - DB? No
  - XML? No
  - demo suitable for QA test? No. 1) Demo includes Risk calculations. 2) Source model contains over 100 sources.

scenario:
  - end-to-end? Yes
  - DB? No
  - XML? No
  - demo suitable for QA test? No. 1) Demo includes Risk calculations.

disaggregation:
  - end-to-end? Yes
  - DB? N/A (no disaggregation results are currently stored in the database)
  - XML? Yes
  - demo suitable for QA test? Yes

UHS:
  - end-to-end? Yes
  - DB? Yes
  - XML? No. (Only file creation is covered; the actual contents are not checked in the current QA tests.)
  - demo suitable for QA test? Yes

[Damiano Monelli (monelli)]

For the event-based QA test I think you can probably still use the demo by removing the risk calculation (this can be simply done by deleting the [risk] section in the configuration file ). In order to reduce the computation time you can also reduce the number of logic tree samples and seismicity histories from 5 to 2 (so that you can still check that results by multiple logic tree samples and seismicity histories are produced. 5 was used in order to have nice risk results but is not needed by the hazard calculations).
If this does not reduce the computation time enough, I can work with you in order to simplify also the source model (to reduce the number of sources).