SRU/Z39.50 slowness when retrieving holdings

[Jeff Davis]

Evergreen 2.10 (but all supported versions are likely affected)
OpenSRF 2.4
Postgres 9.4

When retrieving records with holdings via SRU, gathering the holdings can be quite slow on a large system. Since the Z39.50 service uses SRU under the hood, that means Evergreen can be quite slow to respond to Z39.50 queries as well.

An SRU search like the following (author search for "dorsey" at a specific org unit, with holdings, limited to 10 results) takes over 15 seconds in our production environment:

/opac/extras/sru/BR1/holdings?version=1.1&operation=searchRetrieve&query=eg.author%3Ddorsey&startRecord=1&maximumRecords=10

A similar search in the OPAC returns results in at most 2-3 seconds. The culprit here seems to be the call to open-ils.supercat.record.basic_holdings.retrieve in OpenILS::WWW::SuperCat->sru_search, which can take more than 1 second per record if you have enough records/holdings in your system.

I have a proposed fix for this issue which I'll share momentarily.

[Jeff Davis (jdavis-sitka)]

Fix pushed to working branch user/jeffdavis/lp1609556-z3950-sru-slowness:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=commitdiff;h=0743bcf

This replaces the call to open-ils.supercat.record.basic_holdings.retrieve with a cstore request to unapi.bre. If holdings are requested, we tell unapi.bre to include holdings_xml, and convert the <holdings> element in the response into a set of MARC 852 tags.

Testing this approach on a large system, I see response times in the 2-3 second range (comparable to the OPAC), as opposed to 15+ seconds previously.

[Mike Rylander (mrylander)]

Picked to master for 2.11, but leaving for other RMs to decide on the backport. Seems safe enough to me, though.

[Jason Boyer (jboyer)]

Speaking as someone running this patch on 2.9, I'd say it seems pretty safe too. :)

[Galen Charlton (gmc)]

While evaluating this for backporting to 2.10, I found that the patch had a (unintended?) side-effect: the 852$d now outputs the circ library's full name, not its description. The patch at the tip of the user/gmcharlt/lp1609556_code_for_circ_lib branch restores the previous behavior. I request that it be applied, as clients may rely on particular values in the 852$d.

[Mike Rylander (mrylander)]

Thanks, Galen, great catch! Applied to master.

[Galen Charlton (gmc)]

I've now backported Jeff's patch and my follow-up to rel_2_10.

[Jason Stephenson (jstephenson)]

Should we backport this to rel_2_9? I'm thinking yes, but would like some consensus before I do.

[Galen Charlton (gmc)]

+1 for backporting to 2.9

[Mike Rylander (mrylander)]

+1 from me as well.

[Jason Stephenson (jstephenson)]

Backported it to rel_2_9.