Event Based Calculation - Hazard: Post-Processing Phase takes too long

Bug #1169703 reported by Damiano Monelli
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenQuake Engine
Fix Released
High
Michele Simionato

Bug Description

When running an Event Based Hazard calculation for a realistic national scale calculation, the computation time associated to the 'execution' phase took ~ 1 hour, while the post-processing phase did not finish after ~ 9 hours. I killed the job because it was taking too much time. I report the log here. Attached you can find the input files to reproduce the job. The job was run on the ETH cluster.

[2013-04-16 11:16:25,535 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** pre_executing (hazard)
[2013-04-16 11:16:25,544 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** initializing sources
[2013-04-16 11:17:17,501 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** initializing site model
[2013-04-16 11:17:19,771 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** initializing realizations
[2013-04-16 11:17:21,514 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** executing (hazard)
[2013-04-16 11:19:34,098 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 1% complete
[2013-04-16 11:19:53,704 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 2% complete
[2013-04-16 11:20:07,845 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 3% complete
[2013-04-16 11:20:28,312 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 4% complete
[2013-04-16 11:21:00,087 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 5% complete
[2013-04-16 11:21:41,362 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 6% complete
[2013-04-16 11:21:53,915 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 7% complete
[2013-04-16 11:22:06,092 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 8% complete
[2013-04-16 11:22:28,357 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 9% complete
[2013-04-16 11:23:12,835 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 10% complete
[2013-04-16 11:25:14,485 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 11% complete
[2013-04-16 11:25:49,176 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 12% complete
[2013-04-16 11:26:13,173 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 13% complete
[2013-04-16 11:26:37,375 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 14% complete
[2013-04-16 11:27:05,640 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 15% complete
[2013-04-16 11:27:51,841 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 16% complete
[2013-04-16 11:29:04,238 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 17% complete
[2013-04-16 11:29:55,581 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 18% complete
[2013-04-16 11:30:31,753 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 19% complete
[2013-04-16 11:31:03,333 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 20% complete
[2013-04-16 11:31:38,854 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 21% complete
[2013-04-16 11:32:12,271 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 22% complete
[2013-04-16 11:32:57,839 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 23% complete
[2013-04-16 11:33:45,962 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 24% complete
[2013-04-16 11:34:45,853 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 25% complete
[2013-04-16 11:35:35,337 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 26% complete
[2013-04-16 11:36:09,691 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 27% complete
[2013-04-16 11:36:39,877 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 28% complete
[2013-04-16 11:37:12,392 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 29% complete
[2013-04-16 11:37:58,737 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 30% complete
[2013-04-16 11:38:49,729 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 31% complete
[2013-04-16 11:40:08,773 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 32% complete
[2013-04-16 11:40:50,409 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 33% complete
[2013-04-16 11:41:17,556 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 34% complete
[2013-04-16 11:41:46,734 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 35% complete
[2013-04-16 11:42:20,641 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 36% complete
[2013-04-16 11:43:17,616 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 37% complete
[2013-04-16 11:44:34,526 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 38% complete
[2013-04-16 11:45:16,357 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 39% complete
[2013-04-16 11:45:53,817 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 40% complete
[2013-04-16 11:46:27,111 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 41% complete
[2013-04-16 11:47:01,286 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 42% complete
[2013-04-16 11:47:40,917 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 43% complete
[2013-04-16 11:48:49,904 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 44% complete
[2013-04-16 11:49:43,986 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 45% complete
[2013-04-16 11:50:28,051 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 46% complete
[2013-04-16 11:51:08,183 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 47% complete
[2013-04-16 11:51:43,826 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 48% complete
[2013-04-16 11:52:26,195 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 49% complete
[2013-04-16 11:53:25,248 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 50% complete
[2013-04-16 11:54:21,083 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 51% complete
[2013-04-16 11:55:20,616 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 52% complete
[2013-04-16 11:56:17,950 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 53% complete
[2013-04-16 11:56:55,464 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 54% complete
[2013-04-16 11:57:25,132 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 55% complete
[2013-04-16 11:58:03,489 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 56% complete
[2013-04-16 11:58:50,041 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 57% complete
[2013-04-16 11:59:55,708 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 58% complete
[2013-04-16 12:00:47,616 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 59% complete
[2013-04-16 12:01:33,937 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 60% complete
[2013-04-16 12:02:05,977 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 61% complete
[2013-04-16 12:02:42,018 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 62% complete
[2013-04-16 12:03:21,188 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 63% complete
[2013-04-16 12:03:56,186 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 64% complete
[2013-04-16 12:04:51,346 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 65% complete
[2013-04-16 12:05:35,023 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 66% complete
[2013-04-16 12:06:01,424 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 67% complete
[2013-04-16 12:06:23,619 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 68% complete
[2013-04-16 12:06:51,914 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 69% complete
[2013-04-16 12:07:27,393 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 70% complete
[2013-04-16 12:08:15,494 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 71% complete
[2013-04-16 12:09:02,987 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 72% complete
[2013-04-16 12:09:41,978 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 73% complete
[2013-04-16 12:10:10,199 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 74% complete
[2013-04-16 12:10:38,947 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 75% complete
[2013-04-16 12:11:02,872 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 76% complete
[2013-04-16 12:11:34,932 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 77% complete
[2013-04-16 12:11:52,353 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 78% complete
[2013-04-16 12:12:07,999 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 79% complete
[2013-04-16 12:12:19,662 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 80% complete
[2013-04-16 12:12:37,369 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 81% complete
[2013-04-16 12:12:51,363 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 82% complete
[2013-04-16 12:13:05,438 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 83% complete
[2013-04-16 12:13:14,223 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 84% complete
[2013-04-16 12:13:31,815 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 85% complete
[2013-04-16 12:14:42,310 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 86% complete
[2013-04-16 12:15:01,630 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 87% complete
[2013-04-16 12:15:18,275 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 88% complete
[2013-04-16 12:15:31,712 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 89% complete
[2013-04-16 12:15:51,638 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 90% complete
[2013-04-16 12:16:10,918 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 91% complete
[2013-04-16 12:16:46,495 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 92% complete
[2013-04-16 12:17:24,149 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 93% complete
[2013-04-16 12:17:41,106 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 94% complete
[2013-04-16 12:18:08,999 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 95% complete
[2013-04-16 12:18:29,291 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 96% complete
[2013-04-16 12:18:46,764 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 97% complete
[2013-04-16 12:19:02,241 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 98% complete
[2013-04-16 12:19:30,347 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** > hazard 99% complete
[2013-04-16 12:20:23,467 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** calculation 100% complete
[2013-04-16 12:20:23,475 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** post_executing (hazard)
[2013-04-16 12:20:23,486 #2 gemcontrol.ethz.ch PROGRESS MainProcess/25157 root] ** post_processing (hazard)
[2013-04-16 21:05:32,725 #2 - ERROR MainProcess/25158 supervisor] job process 25157 crashed or terminated

Revision history for this message
Damiano Monelli (monelli) wrote :
Changed in oq-engine:
importance: Undecided → High
milestone: none → 1.0.0
Changed in oq-engine:
assignee: nobody → Michele Simionato (michele-simionato)
status: New → In Progress
Revision history for this message
Michele Simionato (michele-simionato) wrote :

See https://github.com/gem/oq-engine/pull/1144 for a patch that should improve the situation

Changed in oq-engine:
status: In Progress → Fix Committed
Changed in oq-engine:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.