Disaggregation Failure GEM Cluster

[Laurentiu D.]

The following errors occurred during a disaggregation computation in the GEM cluster:
http://pastie.org/7910299
This is a disaggregation computation for 10 sites and the entire SHARE Area Source Model.

[Lars Butler (lars-butler)]

This failure seems to be due to the following reasons:

- PostgreSQL's `shared_buffers` was set to 1024MB
- Each disaggregation matrix INSERT is about ~64MB (in this particular calculation)
- Since each query consumes about 2-3x that, we could easily run max out the buffer if 5-6 workers are inserting simultaneously.

We're going to experiment with the following:
- Increase `shared_buffers` to 12288MB
- Increase `checkpoint_segments` to 64 (from the default of 3). The postgres documentation [1] recommends increasing this in tandem with any increase to `shared_buffers`. Also, we're getting a LOT of checkpoint messages in the postgres log (see below); this should help silence those messages.

LOG: checkpoints are occurring too frequently (26 seconds apart)
HINT: Consider increasing the configuration parameter "checkpoint_segments".

[1] - http://www.postgresql.org/docs/9.1/static/runtime-config-resource.html

[Lars Butler (lars-butler)]

Update:

The configuration changes above did not fix the problem; we're still getting the exact same error: invalid memory alloc request size 1073741824.

I will note that this happens when we try to insert large disagg matrix blobs into the hzrdr.disagg_result table, but it doesn't happen everytime. After the job was aborted, I noticed that 312 records were inserted into this table, so this isn't a consistent failure.

I'm still looking into other options for addressing this issue. I'll post updates as soon as I have them.

[Lars Butler (lars-butler)]

I found the root cause of this error:

We're using a BYTEA field to store the disagg matrices (as pickled numpy arrays), and we're hitting a hard-coded limit to the field size in postgres. The maximum allowed size for BYTEA is 1GB, and we encounter this error when we try to execute an INSERT statement containing ~530MB of data. It seems that a multiplier of 2-3x is required for parsing and processing the data, so that's why we're hitting the limit.

This thread describes a similar problem: http://www.postgresql.org/message-id/[e-mail]

One possible solution (as the thread suggests) is to break up the binary blob into multiple chunks and store them in separate records.

[Michele Simionato (michele-simionato)]

This may not appear anymore. After the change in https://github.com/gem/oq-engine/pull/1390 only the relevant portions of the disaggregation matrix are stored in the database, which means only only few kilobytes, well under the limit of ~530M.