Disaggregation Failure GEM Cluster
Bug #1180274 reported by
Laurentiu D.
This bug affects 1 person
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenQuake Engine |
Fix Released
|
High
|
Michele Simionato |
Bug Description
The following errors occurred during a disaggregation computation in the GEM cluster:
http://
This is a disaggregation computation for 10 sites and the entire SHARE Area Source Model.
description: | updated |
Changed in oq-engine: | |
status: | New → Confirmed |
importance: | Undecided → High |
assignee: | nobody → Lars Butler (lars-butler) |
milestone: | none → 1.0.0 |
Changed in oq-engine: | |
milestone: | 1.0.0 → 1.0.1 |
Changed in oq-engine: | |
assignee: | Lars Butler (lars-butler) → nobody |
Changed in oq-engine: | |
status: | Confirmed → In Progress |
assignee: | nobody → Daniele Viganò (daniele-vigano) |
assignee: | Daniele Viganò (daniele-vigano) → nobody |
assignee: | nobody → Michele Simionato (michele-simionato) |
Changed in oq-engine: | |
status: | Fix Committed → Fix Released |
To post a comment you must log in.
This failure seems to be due to the following reasons:
- PostgreSQL's `shared_buffers` was set to 1024MB
- Each disaggregation matrix INSERT is about ~64MB (in this particular calculation)
- Since each query consumes about 2-3x that, we could easily run max out the buffer if 5-6 workers are inserting simultaneously.
We're going to experiment with the following: segments` to 64 (from the default of 3). The postgres documentation [1] recommends increasing this in tandem with any increase to `shared_buffers`. Also, we're getting a LOT of checkpoint messages in the postgres log (see below); this should help silence those messages.
- Increase `shared_buffers` to 12288MB
- Increase `checkpoint_
LOG: checkpoints are occurring too frequently (26 seconds apart) segments" .
HINT: Consider increasing the configuration parameter "checkpoint_
[1] - http:// www.postgresql. org/docs/ 9.1/static/ runtime- config- resource. html