Search Showing Bibs with no Holdings

[Robert J Jackson]

Hoopla Bib entries are showing for sites that have no subscription.

Performing a search in both the OPAC and Staff Client will result in display of bibs that are set up to only display when a site has license within Hoopla (or other vendor such as Overdrive). The record has valid 856 entry limiting which sites should see record. Record shows in results list but links to Hoopla holdings are not shown. Example from OPAC follows (exand marc record for detail):

http://evergreen.lib.in.us/eg/opac/results?query=Quentin+Tarantino%27s+the+hateful+8.&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=poprel

This appears to have started with code modification in 3.0 for the search engine?

[Robert J Jackson (rjackson-deactivatedaccount)]

Attaching example marc xml

http://evergreen.lib.in.us/eg/opac/record/20647723?query=Quentin%20Tarantino%27s%20the%20hateful%208.;qtype=keyword;locg=1;detail_record_view=0;sort=poprel;expand=marchtml#marchtml

[Kathy Lussier (klussier)]

Confirming that this is new behavior in 3.0. I did side-by-side comparisons between a master system and the MassLNC 2.12 community demo server, and the problem does not occur on 2.12.

I added to records to the database, one with a located URI set to CONS and one with a located URI set to SYS1.

In the staff client, both records are retrieved no matter what the scope of the search. If I scope the search to CONS, I should not retrieve the SYS1 record, but it is retrieved. If I scope the search to SYS2, both records appear.

On the other hand, in the public catalog, no records with Located URIs are retrieved, no matter what the scope of the search.

[Kathy Lussier (klussier)]

I'm just adding a couple of notes of clarification:

1. Robert and I are seeing different behavior in the OPAC searches. In his case, he retrieves the same record in the OPAC and in the staff client. In my case, I retrieved all the located URI records in the staff client, but was unable to retrieve any in the OPAC.

2. Looking at this IRC log here - http://irc.evergreen-ils.org/evergreen/2017-09-01#i_321791 - I think we see another instance where the OPAC search was not retrieving records that it should be retrieving.

3. I also enabled the 'Located URIS should behave like copies' global flag. I retrieved / did not retrieve the same bib records no matter what the state of that setting.

[Kathy Lussier (klussier)]

After talking about this in IRC, it looks like bug 1730758 may be the cause of the issue I am seeing. However, I'm hesitant to mark this bug as a duplicate since the behavior Robert is seeing for OPAC searches is a little different and may a separate bug.

[Mike Rylander (mrylander)]

I believe that the branch on bug 1730758 may address both of the issues reported here. Specifically, I'm 99% sure about Kathy's, but I'm less sure about Robert's.

[Jeff Davis (jdavis-sitka)]

I applied the fix from bug 1730758 on a test server running 3.0.2. The test dataset has four ebook records for works by J.R.R. Tolkien with located URIs and no physical holdings. Previously the vis_attr_vector field was null for those four records; after running the upgrade script, that field was populated.

However, the four ebook records still aren't being retrieved in a keyword search for "tolkien" in the public OPAC. I tried adding more located URIs (BR1 in addition to CONS) and searching at various scopes (CONS, SYS1, BR1) with no success. The records are not retrieved regardless of whether the opac.located_uri.act_as_copy flag is enabled.

[Michele Morgan (mmorgan)]

I did some testing on our 3.0.1 upgraded system.

For bib records that existed prior to the upgrade, visibility behaves as expected in both the web staff client and the public catalog:

- A record with a located uri is retrieved in the scope of the located uri, but not in the scope of another branch.

Edited bib records that existed prior to the upgrade also behave as above.

For a newly added record with a located uri:

- In the web staff client, the record is retrieved in the scope of the located uri. The record is also retrieved (without the link) in the scope of another branch.

- In the public catalog, the record is not retrieved in any scope.

[Mike Rylander (mrylander)]

Michele, was your testing with or without my patch above? Thanks!

[Michele Morgan (mmorgan)]

My testing for comment 9 was on an unpatched system. I have since applied the patch and am now seeing the following for a record with a located uri:

In the staff client:

- The record is retrieved in the scope of the located uri
- The record is retrieved in the cons scope
- The record is retrieved in another branch's scope but the link does not display

In the public catalog:

- The record is retrieved in the scope of the located uri
- The record is retrieved in the cons scope
- The record is not retrieved in another branch's scope

This behavior is the same for existing, new and edited records.

The problem I see with the patched behavior is the staff client behavior where the record appears in a scope that does not match the located uri. Otherwise, the visibility is behaving as expected.

We have the opac.located_uri.act_as_copy flag enabled.

[Kathy Lussier (klussier)]

I'm adding the results of my testing on a master system loaded with Mike's patch from bug 1730758. This system has a new database with Concerto data. The system also has two new bib records with Located URIs.

There is a small amount of improvement for staff client located URI searches, but I still see significant issues.

* In the public catalog, I am still unable to retrieve any records with Located URIs.
* In the staff client (I was using the web client), the one area where I see improvement is if I am searching totally outside the scope for the located URI's owner (e.g. searching BR3 for a record where the Located URI was owned by BR1), the record is no longer retrieved.
* I retrieve the same results regardless of the state of the opac.located_uri.act_as_copy flag.
   - When the scope is set to the consortium, I retrieve records owned where the located URI is owned by child org units. (unexpected behavior when opac.located_uri.act_as_copy is set to FALSE, expected behavior when opac.located_uri.act_as_copy is set to True).
   - When the scope is set to a system and the located URI is set to a child org unit, I do not retrieve the record (expected behavior when opac.located_uri.act_as_copy is set to FALSE, unexpected behavior when opac.located_uri.act_as_copy is set to True.)

I have not tested this patch on a system with an upgraded database, and I believe Michele's results are a bit different than mine.

This bug is a critical for our libraries. As a result of this bug and bug 1736992, we have postponed a 3.0 upgrade scheduled for next week since these search issues would be a significant regression for users.

[Mike Rylander (mrylander)]

Kathy or Michele, would you be able to provide the SQL-level search query for each situation? TIA.

[Kathy Lussier (klussier)]

Thanks for looking into this Mike! I have a few queries from the master system that has the patch loaded from bug 1730758.

With the opac.located_uri.act_as_copy flag set to FALSE,

Query for public catalog search that does not retrieve results but should retrieve a record with a Located URI (scope and Located URI owner match) - https://pastebin.com/nfnZ6kez

Query for staff catalog search scoped to the consortium that does retrieve a record when it should not (scope set at CONS, Located URIs owned by child branches) - https://pastebin.com/5RG99YBf

With the opac.located_uri.act_as_copy flag set to TRUE,

Query for public catalog search that does not retrieve results but should retrieve a record with a Located URI (scope set to CONS, Located URIs owned by child branches) - https://pastebin.com/EMJPF3bs

Query for staff catalog search that does not retrieve results but should retrieve a record with a Located URI (scope set to SYS, Located URI owned by child branch) - https://pastebin.com/5RG99YBf

[Mike Rylander (mrylander)]

Please find a fix for these, as well as bug 1736992, at:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1736419-located-uri-search-and-available

From the commit message:

The site() filter and #staff modifier are used to decide when and how to
include certain query filters, such as circ_lib or luri_org. Unfortunately,
site() is sometimes excluded (whole-tree search) and the test for staff-
iness was not specific enough, leading to incorrect queries in those cases
where information was lacking. Now, we treat site() specially, forcing a
default of "top-of-tree", and we check for the #staff modifier directly
where necessary.

Note: this commit also addresses LP#1736992 which is about staff searches
using the limit-to-available modifier. As a side effect of the special
site() treatment, that issue is resolved.

[Jeff Davis (jdavis-sitka)]

Installing the branch failed for me on "grunt all", I think for reasons unrelated to this issue. So I tried applying just Mike's commit on a clean 3.0.2 install with the luri-act-as-copy flag enabled. I also applied that commit separately on a test server that had previously been upgraded to 3.0.2. In both cases, unfortunately, I saw no change in behavior: after a full restart of services, titles with located URIs (both existing records in the test dataset and newly created ones) still do not show up in public search results at any scope.

[Jeff Davis (jdavis-sitka)]

Attached is the Postgres search query for "tolkien" (scoped to CONS) on a clean 3.0.2 install with Mike's patch applied. This search should return several records with located URIs which are in the test data, but it returns 0 results.

[Kathy Lussier (klussier)]

Adding a note that I installed on a clean master install with this patch and the patch from bug 1730758 installed. I also am unable to retrieve records with the tolkien search in the public catalog. Records are retrieved in the staff client.

I'm going to add some more records now to see if the other staff client searches have improved.

[Mike Rylander (mrylander)]

Ah, ok, so everyone here is testing with "act_as_copy" enabled, but without a truthy value set. The code in 3.0 looks for a truthy value, though. Looking back at the old query_parser_fts function, it was just checking to see if the "enabled" boolean was set to true. I have updated the above branch to do what the old code did, and ask that you test with that new change in place.

Thanks, all!

[Mike Rylander (mrylander)]

As a side note, would someone mind testing with "act_as_copy" disabled? I'm testing that way and seeing exactly the expected behavior.

Thanks!

[Kathy Lussier (klussier)]

Adding more notes from my testing on the clean master install with the two Mike patches:

- The limit by available filter appears to be working now in both the staff client and the catalog.
- As Jeff previously noted, I'm still unable to retrieve any records with a Located URI in the public catalog.
- For staff client, Located URI searches, I added a record with a Located URI owned by BR1. With the opac.located_uri.act_as_copy set to FALSE, I was able to retrieve results as expected. No results for searches scoped to CONS, SYS1 or any of the SYS2 org units. The record was successfully retrieved when set to a BR1 scope.
- However, the staff client search did not appear to recognize the opac.located_uri.act_as_copy flag. When I set it to TRUE, I was unable to retrieve this record at the CONS or SYS1 scopes as expected.

Thanks for your work thus far Mike!

[Kathy Lussier (klussier)]

Also, I have the full SQL query for the staff client search where the opac.located_uri.act_as_copy flag was set to TRUE.

https://pastebin.com/373xzrpk

[Kathy Lussier (klussier)]

Sorry Mike! I missed your earlier comment before posting my follow up. However, I want to clarify something. Most of my testing was done with act_as_copy set to FALSE. The public catalog searches fail to retrieve Located URI records no matter what the state of that setting.

[Kathy Lussier (klussier)]

I loaded the branch with Mike's latest changes this morning.

In my testing, the staff client searches for Located URI bibs and using the 'limit to available' filter appear to work as expected, with and without the global flag enabled.

However, the public catalog searches still do not retrieve results with a Located URI. The core query from a search that was run when the global flag was disabled is here - https://pastebin.com/WRtyWDGq

The keyword search is for tolkien and is scoped to the consortium. The Located URIs in the Tolkien records are all owned by the consortium.

The tests were run on a master system with Mike's branches from this bug and bug 1730758 loaded.

[Mike Rylander (mrylander)]

I'm making progress. First, one needs to apply the XXXX upgrade script from bug 1730758 just to get the visibility vector to update correctly. Now I'm dealing with a staff-search issue on non-CONS searches. More soon, I hope.

[Mike Rylander (mrylander)]

With one more change on this branch, and the fixes supplied for bug 1730758, I am getting the expected results for both public and staff, and for either enabled value of opac.located_uri.act_as_copy. I believe this is ready now, and you can see the results at https://dev-test2.esilibrary.com which has the Tolkien records, one of which has had its LURI moved to BR4. The global flag is currently disabled.

[Kathy Lussier (klussier)]

Thanks Mike! I'll add these patches to one of my test systems now.

We are tracing down another problem we've discovered in the wild with the limit by available filter. We've seen cases where using the filter in the public catalog does not remove all unavailable results from the result list. I've seen it occur on egtraining.noblenet.org and in Evergreen Indiana's catalog, both of which are running 3.0, but I have not been able to replicate it on one of my test servers. I tried replicating it on a clean install and on a system upgraded from 2.12, but I couldn't make the problem happen.

I do not think it's related to the biblio.record_entry.vis_attr_vector problem I mentioned in bug 1730758 this morning because there have been no changes to the bib source since the upgrade. You can see the problem happening with results 1 and 4 here - https://egtraining.noblenet.org/eg/opac/results?query=patterson+die&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=&modifier=available - and with result 1 here - http://evergreen.lib.in.us/eg/opac/results?query=danielle+steel+fall+grace&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=poprel&modifier=available

If all of the other issues identified in this bug are fixed with your patch, maybe it would be better to open a new bug on this issue.

[Mike Rylander (mrylander)]

Yes, that's almost certainly a separate issue. Copy visibility is controlled by a separate table, though the same trigger is used to maintain that other table. So, even if it's the trigger that is touched by the fix on bug 1730758, it'll be a new and different fix.

Thanks, Kathy!

[Mike Rylander (mrylander)]

Marking this as a duplicate of bug 1736419 which, while not strictly true, is the best LP can do. The fix for that bug addresses the issue here as a side effect of making the code more correct over there.

[Mike Rylander (mrylander)]

Arg ... ignore comment #29. That was meant for the available-modifier bug 1736992.

[Jeff Davis (jdavis-sitka)]

Searching for "tolkien" in the public catalog retrieves the expected located URI results for me now on a 3.0.2 test system with the commit from bug 1730758 and the three commits for this bug. I'll do some further testing but it looks like that issue is fixed. Thanks, Mike!

[Kathy Lussier (klussier)]

This looks good to me too. I tested on a clean master system and on an upgraded database. Records are being retrieved as expected.

I signed off on the branch and merged it to master and release 3.0.

Thank you Mike, Jeff, Robert and Michele!