Bibs with Transcendant of 'f' showing on location limited searches

[Robert J Jackson]

web client 3.0.3

search:
keyword "massachusetts"
Shelby County PL - Shelbyville Main Library
Limited to "All Books"
Shelving Location="J Nonfiction"

Included in the hits are GPO - Government Documents which have the following settings in the config.bib_source table:

evergreen=# select * from config.bib_source where id = 9;
 id | quality | source | transcendant | can_have_copies
----+---------+----------------------------+--------------+-----------------
  9 | 90 | GPO - Government Documents | f | t

Attaching a screen shot of the search screen showing one of several non transcending records.

[Mike Rylander (mrylander)]

Unless I'm misunderstanding you, this is exactly as expected -- they're entirely empty records, so they're supposed to show up in staff searches. Otherwise, staff would not be able to find them to catalog items or add LURIs. Am I missing something?

[Elaine Hardy (ehardy)]

That is expected behavior for the reasons you state. It is a feature and not a bug.

[Elaine Hardy (ehardy)]

Provided it is the staff client and not the public....

[Robert J Jackson (rjackson-deactivatedaccount)]

Sorry - Should never have made it launch pad so it seems. Am I mis-remembering or was there talk in the past to add an option to exclude non-transcending bibs with no holdings from results in the staff client?

[Kathy Lussier (klussier)]

If this is a public search, I believe this could be similar to bug 1745233. We're still seeing issues with copy location searching here.

[Elaine Hardy (ehardy)]

As long as it is an option. PINES catalogers depend on being able to find records with no holdings attached in a staff client search. At one time, there was a color cue (the record was highlighted in gray in the retrieved search list) for those records so that staff could easily tell what they were, lessening confusion for circ staff. We would like that color coding back.

[Jason Boyer (jboyer)]

Elaine, do you know if it's common in PINES to limit searches to a specific shelving location while expecting to see these records? That seems to be the only time we hear of staff confusion and hiding empty records when searching at a copy location would be a smaller change than trying to add an option to enable / disable empty records in searches altogether.

[Elaine Hardy (ehardy)]

I haven't heard of staff confusion in years -- at least, I haven't seen helpdesk tickets or had questions sent to me about them in a very long time. I think staff in PINES are aware enough of the empty records that they know what they are. To me, it is a matter of staff education that would be assisted by having that color cue back.

[Kathy Lussier (klussier)]

In my mind, when you limit by copy location, you're trying to find materials that have had that copy location applied to it. If the record has no copies, then the copy location has not been applied. I would support removing empty records from copy location limited searches.

[Elaine Hardy (ehardy)]

I don't see catalogers filtering by copy location very often when looking for an existing record to attach holdings, so I don't see removing empty records from those results to hamper cataloging workflow. Perhaps if they thought the title was already in their holdings, but I don't think they would expect to retrieve an empty record in that case.

[Beth Longwell (blongwel)]

Limiting a search down to a specific shelving location should not return empty records. While I try to keep these cleaned up, there are always bibs waiting for items, and it is confusing to staff and patrons. Our current Evergreen version is 3.1.11

[Michele Morgan (mmorgan)]

Adding a long overdue Confirmed status to this.

Still an issue in Evergreen 3.7

While it's understandable that bibs with no items should display in staff catalog searches so that holdings can be attached, Comment #10 states the real issue succinctly:

"when you limit by copy location, you're trying to find materials that have had that copy location applied to it. If the record has no copies, then the copy location has not been applied."

Empty bibs should not show in searches limited to shelving locations or shelving location groups.

[Lindsay Stratton (lstratton)]

Complete agreement that records with no holdings should not appear when limiting a search by shelving location.

(On a related note - empty bibs are also retrieved when "limit to available" is selected... https://bugs.launchpad.net/evergreen/+bug/1745446)

[Mike Rylander (mrylander)]

Would it be acceptable to have a mechanism for turning /off/ "empty record inclusion" on demand in any staff search, rather than changing the default behavior?

Or should, specifically and only, location (and location group) and #available searches disable empty records?

[Lindsay Stratton (lstratton)]

IMO, specifically location/location group and available item searches should exclude empty records.

Most people using those types of searches are doing so with the expectation that the search will return actual holdings.

[Elaine Hardy (ehardy)]

Specifically location/location group and available search.

Still advocating for the return of the color difference for empty bibs.

[Jeff Godin (jgodin)]

My understanding is that the use case for showing records with no holdings is for staff to be able to find these records in order to attach holdings (or located URIs).

I suspect that most staff users and most staff searches do not benefit from having these empty records included in search results. (I don't have hard data to back up this suspicion.)

My vote would be for these "empty" bibs to not appear in staff searches unless a specific option is passed to the search.

To ease the change in default behavior, staff search interfaces could consult a user or workstation setting, and pass the new option in searches automatically.

Even if the default behavior is based on a stored setting, I think that the option should be exposed in the normal staff search UI as well. This will permit occasional use, as well as make it clear that the option is on/off.

I don't think that the UI option should be "sticky" -- it shouldn't change the stored setting. The stored setting could be changed from the "Catalog Preferences" interface.

Some search interfaces (bucket search) might *always* include empty records.

An upgrade script (perhaps just a suggested script/query) could set this new "include empty records" stored setting based on a specific user permission, certain groups, etc.

Another option would be to discard the idea of a stored setting and a UI option, and ONLY show empty records in staff search results if the user performing the search has any of a list (to be determined) of permissions indicating that they can attach holdings / add located URIs.

In any case, a shading of empty records also sounds like a good idea, independent of the ideas above.

[Mike Rylander (mrylander)]

I've pushed a WIP branch to https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1746800-exclude_empty_bibs_option which adds an #exclude_empty search modifier. When included in a staff search, this disables the "include empty bibs" logic. It would be simple to flip the logic to an #include_empty modifier, but this avoids changing the current default behavior while allowing the UI to present the option to staff.

[Dan Briem (dbriem)]

Piggy-backing on Mike's direction, this branch is from the New Developers Working Group:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbriem/lp1746800_exclude_empty_records_on_filtered_staff_searches

It excludes empty bibs on staff searches when filtering on location or location group or when using the availability modifier.

[Terran McCanna (tmccanna)]

See also, related bugs:
https://bugs.launchpad.net/evergreen/+bug/1929072
https://bugs.launchpad.net/evergreen/+bug/1664585

[Jennifer Weston (jweston)]

Tested on terran-master during feedback fest

Looks good!

Functionality behaves as described by Dan with the branch from New Dev group.
Searched scenarios:
#1 a)keyword=music, scope=CONS; returns 107 results, including 1 empty bib (tcn 34)
#1 b)keyword=music, scope=CONS; limit to available; returns 106 results, does not include empty bib
#1 c)keyword=music, scope=CONS; loc=Stacks; 106 results, does not include empty bib
#1 d)keyword=music, scope=BR3; 107 results, including 1 empty bib (tcn 34)
#1 e)keyword=music, scope=BR3 loc=Music; 10 results, does not include empty bib
#2 Repeated again with keyword = science and same iterative steps; behaved as expected ignoring empty bib if specific location or limit to available chosen

Does not address scenario from bug 1929072 where 3 or more keywords input does not honor location limit (but that is documented in a separate bug)

I would sign off but I have one question -- on comment #19 Mike refers to a new #exclude_empty search modifier; I do not see that. The functionality is working without it per testing notes above. If no additional search modifier is expected, then I am comfortable signing off.

[Terran McCanna (tmccanna)]

Hi Jennifer,

The New Devs Group looked at Mike's example with the new exclude_empty modifier but realized we could leverage the same idea just by checking the existing options rather than creating the new option.

Thanks for testing!

[Jennifer Weston (jweston)]

Thanks, Terran! I just verified with Mike, too, that I should not be seeing a modifier.

I'll sign-off. I have tested this code and consent to signing off on it with my name, Jennifer Weston and my email address, <email address hidden>

[Mike Rylander (mrylander)]

Thanks, all! I've merged Dan's branch into master for 3.9. I did not backport it because it is a behavioral change, and one that (while consensus is to change it) folks are familiar with in released version.

[Dan Briem (dbriem)]

Thanks, Mike.

Are you open to reconsidering backporting this?

I think your reasoning makes sense, but when I think about how many systems will not be on 3.9 for some time, I'm concerned that service desk workers searching quickly for filtered staff-side info by available items or shelving location do not have an alternative. Staff looking for empty records can just not use these filters as they are not relevant to those searches.

I'm also concerned about new or prospective (or existing) Evergreen users potentially drawing inaccurate conclusions about the quality of search results because they might not realize what's happening.

[Michele Morgan (mmorgan)]

FWIW, I would support backporting this. It is a behavioral change for sure, but it changes unexpected behavior to expected behavior.

Discussion on this bug, bug 1929072 and bug 1664585 mentioned in comment #21, and the general list discussion referenced in 1929072:

http://list.evergreen-ils.org/pipermail/evergreen-general/2021-May/000524.html

all reinforce that the current behavior is unexpected and leads to confusion.

[Mike Rylander (mrylander)]

Who am I to stand in the way of consensus? :)

Merged to rel_3_7 and rel_3_8 for maintenance release goodness.

Thanks, all!

[Mary Llewellyn (mllewell)]

Coming in late to this...Probably too late to affect the development, but figured I'd add our 2 cents anyway.

Our public services staff expects that when they limit a search to their library, they should only see bib records with their items attached.

They are generally not adding in shelving location to filter their searches.
Limiting to Available copies is not helpful if they are looking to place holds on items out in circulation.

Our catalogers are not limiting searches to their library. They search CONS so they can find all records, with and without their items. It is not useful in our consortium for our catalogers to limit searches to their libraries in hopes of finding a record with no items. Once one library has added an item, that type of search will not be fruitful for the next library.

I am hoping for a setting where we could suppress retrieving empty bibs when searching by library, and only allow them to appear in a CONS search.

[Elaine Hardy (ehardy)]

In the PINES consortium, some of our libraries have cat1s that import the records and cat2s that add holdings, so they do need to find a record with no holdings attached if they do limit the search to their library.

Having a check box that filters for only records with attached holdings would be a solution, I think.

[Mike Rylander (mrylander)]

While I think that the code created by the new-devs group is a good feature, I agree that having a way to explicitly exclude "empty" records would be another good improvement. That was the intent of my branch above. I'll open a new bug for that, specifically, and link to my earlier branch as a starting point. It will need a UI component if we want to be able to check a box, but it would make such a filter accessible through the search grammar even in the state it exists now.

[Mary Llewellyn (mllewell)]

If a check box is added, could it be made sticky?

[Andrea Neiman (aneiman)]

updated LP to reflect this is in rel_3_7 and rel_3_8