oq-engine: better organization of hazard output results

Bug #1163276 reported by Damiano Monelli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenQuake (deprecated)
Fix Released
High
Lars Butler

Bug Description

To facilitate users in browsing hazard results generated by oq-engine, we need to better organize the hazard results when they are serialized to xml.

For each calculation a folder has to be generated, with the following structure:

- job_id:
             - hazard_result_type1:

                    - intentisity_measure_type1:

                            - hazard_result_logic_tree_id1.xml
                            - hazard_result_logic_tree_id2.xml
                            
                            - hazard_results_mean.xml
                            - hazard_results_quantile_quantile_level1.xml
                            - hazard_results_quantile_quantile_level2.xml

                    - intentisity_measure_type2:

                            - hazard_result_logic_tree_id1.xml
                            - hazard_result_logic_tree_id2.xml
                            
                            - hazard_results_mean.xml
                            - hazard_results_quantile_quantile_level1.xml
                            - hazard_results_quantile_quantile_level2.xml

             - hazard_result_type2:
                            

             - hazard_result_type3:
                            

For instance, for a classical PSHA calculation that requires hazard curves, hazard maps and uhs the output folder should look like (considering a source model logic tree containing a single source model with ID = 1, and a GSIM logic tree with two gmpes for a single tectonic region type with ID=1 and ID=2):
- job_1:
             - hazard_curves:
                     - PGA:
                                          - hazard_curves_1_1.xml
                                          - hazard_curves_1_2.xml
                                          - hazard_curves_mean.xml
                                          - hazard_curves_quantile_0.1.xml
                                          - hazard_curves_quantile_0.9.xml
                     - SA-0.1:
                                          - hazard_curves_1_1.xml
                                          - hazard_curves_1_2.xml
                                          - hazard_curves_mean.xml
                                          - hazard_curves_quantile_0.1.xml
                                          - hazard_curves_quantile_0.9.xml
             - hazard_map:
                     - PGA:
                                          - hazard_map_1_1.xml
                                          - hazard_map_1_2.xml
                                          - hazard_map_mean.xml
                                          - hazard_map_quantile_0.1.xml
                                          - hazard_map_quantile_0.9.xml
                     - SA-0.1:
                                          - hazard_map_1_1.xml
                                          - hazard_map_1_2.xml
                                          - hazard_map_mean.xml
                                          - hazard_map_quantile_0.1.xml
                                          - hazard_map_quantile_0.9.xml
             - uhs:
                                          - uhs_1_1.xml
                                          - uhs_1_2.xml
                                          - uhs_mean.xml
                                          - uhs_quantile_0.1.xml
                                          - uhs_quantile_0.9.xml

For a event based calculation requiring stochastic event sets and ground motion fields and hazard curves and hazard maps (as computed from ground motion fields):
- job_1:
             - stochastic_event_sets:
                     - stochastic_event_set_1_1.xml
                     - stochastic_event_set_1_2.xml
                     - complete_logic_tree_stochastic_event_set.xml
             - ground_motion_fields:
                     - ground_motion_fields_1_1.xml
                     - ground_motion_fields_1_2.xml
                     - complete_logic_tree_ground_motion_fields_set.xml
             - hazard_curves:
                     - PGA:
                           - hazard_curves_1_1.xml
                           - hazard_curves_1_2.xml
                           - hazard_curves_mean.xml
                           - hazard_curves_quantile_0.1.xml
                           - hazard_curves_quantile_0.9.xml
                     - SA-0.1:
                           - hazard_curves_1_1.xml
                           - hazard_curves_1_2.xml
                           - hazard_curves_mean.xml
                           - hazard_curves_quantile_0.1.xml
                           - hazard_curves_quantile_0.9.xml
             - hazard_map:
                     - PGA:
                           - hazard_map_1_1.xml
                           - hazard_map_1_2.xml
                           - hazard_map_mean.xml
                           - hazard_map_quantile_0.1.xml
                           - hazard_map_quantile_0.9.xml
                     - SA-0.1:
                           - hazard_map_1_1.xml
                           - hazard_map_1_2.xml
                           - hazard_map_mean.xml
                           - hazard_map_quantile_0.1.xml
                           - hazard_map_quantile_0.9.xml

For a disaggregation calculation:
- job_1:
         - disaggregation_matrices:
                   - PGA:
                         - disaggregation_matrices_1_1.xml
                         - disaggregation_matrices_1_2.xml
                   - SA-0.1:
                         - disaggregation_matrices_1_1.xml
                         - disaggregation_matrices_1_2.xml

For a scenario calculation:
- job_1:
         - ground_motion_fields:
                    - PGA:
                         - ground_motion_fields.xml
                    - SA-0.1:
                         - ground_motion_fields.xml

In the case of Monte-Carlo sampling of the logic trees, it's possible that there will be multiple files with the same branch IDs for a given IMT. For example:

- job_1:
             - hazard_curves:
                     - PGA:
                                          - hazard_curves_b1_b2-ltr0.xml
                                          - hazard_curves_b1_b2-ltr1.xml
                                          - hazard_curves_b1_b3-ltr0.xml
                                          - hazard_curves_b1_b3-ltr1.xml

In this case, we can include "-ltrN", where N is the `ordinal` of the hzrdr.lt_realization record for the output. (See https://github.com/gem/oq-engine/blob/2e57fda8b6568f3021321dffb9ede53a2e5a8cbe/openquake/engine/db/models.py#L2370.)

The same approach can be applied to the other outputs as well (not just hazard curves).

description: updated
description: updated
description: updated
Changed in openquake:
assignee: nobody → Lars Butler (lars-butler)
status: New → In Progress
milestone: none → 1.0.0
description: updated
Changed in openquake:
importance: Undecided → High
description: updated
description: updated
Revision history for this message
Lars Butler (lars-butler) wrote :

Whiteboard photo, detailing all of the various cases for each hazard output type.

Revision history for this message
Lars Butler (lars-butler) wrote :
Changed in openquake:
status: In Progress → Fix Committed
Revision history for this message
Lars Butler (lars-butler) wrote :

A correction to the disagg filename: https://github.com/gem/oq-engine/pull/1137

Changed in openquake:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.