Gateway timeout on retrieving many shelving locations

[Jeff Davis]

EG 2.12.1 and 3.0.3
OpenSRF 2.5 and 3.0

When requesting shelving locations (open-ils.circ.copy_location.retrieve.all) via the HTTP gateway, the request can time out prematurely if you have a large number of shelving locations.

To reproduce, add a large number of shelving locations to asset.copy_location (over 5,000 in my test environment). Then, request all shelving locations via the gateway:

https://example.com/osrf-gateway-v1?service=open-ils.circ&method=open-ils.circ.copy_location.retrieve.all

On my test server, this returns a null response:

{"payload":[],"status":200}

The open-ils.circ.copy_location.retrieve.all API call itself is not failing: OpenSRF logs show no errors, and the request succeeds via srfsh. However, the gateway logs show a timeout error:

Returning NULL from app_request_recv after timeout: open-ils.circ.copy_location.retrieve.all [null]

The default timeout value is 60s, but we get our null response in only a few seconds. The gateway request succeeds if we supply a larger timeout value as a URL param, e.g.:

https://example.com/osrf-gateway-v1?service=open-ils.circ&method=open-ils.circ.copy_location.retrieve.all&timeout=300

So far I haven't been able to replicate the issue with other API calls that return large chunked responses, but I don't know why there would be anything special about open-ils.circ.copy_location.retrieve.all.

[Jeff Davis (jdavis-sitka)]

See attachment for osrfsys and gateway logs for a failed request.

[Jeff Davis (jdavis-sitka)]

Streaming the API response seems to fix the issue:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/jeffdavis/lp1754164-copy-location-gateway-timeout

Compare bug 1756142.

[Bill Erickson (berick)]

Beware, changing an API from non-streaming to streaming changes how the client interacts with the call. Any clients (e.g. Open-ILS/web/js/dojo/openils/CopyLocation.js) that call the API will need to be taught to expect a stream instead of a single array of things.

Also, while we're addressing the issue of fetching too-large data blobs, I suggest we add {substream => 1} to the underlying cstore editor call so it also fetches the big list of stuff via streaming call from cstore.

Removing pullrequest until at least the first issue is resolved.

[Jeff Davis (jdavis-sitka)]

Thanks for reviewing, Bill. I should have known it wouldn't be so easy. :)

I don't know what is required to handle a streaming response from this API. Any assistance would be appreciated.

[Galen Charlton (gmc)]

Noting in discussion with Jeff that it was a non-Evergreen client that had been running into this issue.