located URIs appear in staff client OPAC searches regardless of $9's

[James Fournie]

EG 2.0.10
Postgres 8.4
Ubuntu Lucid

The located URIs seem to appear (as greyed out records) regardless of scoping within the staff client OPAC (but not the public OPAC).

I can't really think of a use case for this -- if I'm searching with my OPAC set to BR1 and "This Branch", I don't want to see 856s with a $9 BR2 under that circumstance -- I'd scope out if I wanted to see them.

~James

[Beth Longwell (blongwel)]

I'm getting a lot of complaints from staff about this display issue as more and more ebooks are added to the catalog. In some searches there are 10 times as many non-owned resources showing up in comparison to owned resources.

Beth Longwell - Sage Library System

[Dan Scott (denials)]

As I understand it, this is not restricted to bibs with located URIs and no copies. In the staff client, _any_ active, non-deleted bib record that matches the search terms will be displayed. Those that don't have copies in the search scope get greyed out.

This behaviour is by design. The use case is that you might want to add a copy or a located URI to one of those bibs, and how else would you find the bibs if you couldn't search for bibs that had no copies or located URIs or transcendence in any scope?

[Jason Etheridge (phasefx)]

On Tue, May 22, 2012 at 9:17 PM, Dan Scott <email address hidden> wrote:
> As I understand it, this is not restricted to bibs with located URIs and
> no copies.  In the staff client, _any_ active, non-deleted bib record
> that matches the search terms will be displayed. Those that don't have
> copies in the search scope get greyed out.

This doesn't seem to be the case for me. In a master version of EG, I
created 3 bibs, titled purple 1, purple 2, and purple 3.

I gave purple 1 one item at BR1.
I gave purple 2 one item at BR2.
I gave purple 3 no items.

My workstation library is BR1, so my staff OPAC defaults to Branch scope at BR1.

I search for purple and I get purple 1 and purple 3 (grayed out). No
sign of purple 2. If that's the behavior we want to keep, I'd expect
for OU-targeted URI's to behave the same as items.

With the TPAC, my default is Consortium, so I see all of them. In my
mind, searching at Consortium is the solution to the visibility
problem. I'm tempted to say we should make it where all grayed out
bibs only show up when searching at the Consortium.

[Beth Longwell (blongwel)]

Dan,

While having these bibs with located URIs display in an unscoped view of
the catalog is appropriate for the reasons you suggested, they should not
appear in a search limited to a specific library. In a recent example
provided by one of our libraries a search for Spanish Civil War, limited to
their library, produced 46 results, only 6 of which belonged to them. When
trying to help a patron find something locally or make a collection
assessment all the irrelevant bibs get in the way and tend to make them
cranky. My feeling is that URI located bibs should be treated like bibs
with copies in that they appear in scopes limited to the owning library or
in the unscoped view.

Beth

On Tue, May 22, 2012 at 6:17 PM, Dan Scott <email address hidden> wrote:

> As I understand it, this is not restricted to bibs with located URIs and
> no copies. In the staff client, _any_ active, non-deleted bib record
> that matches the search terms will be displayed. Those that don't have
> copies in the search scope get greyed out.
>
> This behaviour is by design. The use case is that you might want to add
> a copy or a located URI to one of those bibs, and how else would you
> find the bibs if you couldn't search for bibs that had no copies or
> located URIs or transcendence in any scope?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/925776
>
> Title:
> located URIs appear in staff client OPAC searches regardless of $9's
>
> Status in Evergreen - Open ILS:
> New
>
> Bug description:
> EG 2.0.10
> Postgres 8.4
> Ubuntu Lucid
>
> The located URIs seem to appear (as greyed out records) regardless of
> scoping within the staff client OPAC (but not the public OPAC).
>
> I can't really think of a use case for this -- if I'm searching with
> my OPAC set to BR1 and "This Branch", I don't want to see 856s with a
> $9 BR2 under that circumstance -- I'd scope out if I wanted to see
> them.
>
> ~James
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/925776/+subscriptions
>

[Dan Scott (denials)]

Jason: I'm tempted to say that we should entirely eliminate the weird modal distinction between staff and public OPAC usage and just add a checkbox that says "Show me all the bibs in the system, damn the torpedos"; if that's not checked, then the behaviour of the public OPAC and staff OPAC should be identical in terms of scoping and what is or is not shown.

That way, staff see what the public see, no confusion.

[Jason Etheridge (phasefx)]

> Jason: I'm tempted to say that we should entirely eliminate the weird
> modal distinction between staff and public OPAC usage and just add a
> checkbox that says "Show me all the bibs in the system, damn the
> torpedos"; if that's not checked, then the behaviour of the public OPAC
> and staff OPAC should be identical in terms of scoping and what is or is
> not shown.

Yeah, I like that better.

[Ben Shum (bshum)]

Going back to James' original bug report, I think we can actually confirm that while we were still on 2.0, the located URI's behaved strangely for us. For whatever reason, the subfield 9 acted oddly when used at the library level vs. the system level. In our system, we have a three tier structure (CONS, SYS, BR) and subfield 9 entries at BR didn't behave as expected to restrict to only that BR for display while subfield 9 entries at SYS did; as far as restricting view of said located URIs.

I think this behavior began to change in 2.1+ to match expectations but whatever fixes there were never backported fully to 2.0. And since we had altered all our located URI's anyways, we didn't get to figuring out the actual problem itself.

Will try looking up more details when I get the chance (though 2.0 is dead to me!)

[Beth Longwell (blongwel)]

We are on 2.1 and seeing the same behavior as reported on 2.0.

Beth

On Wed, May 23, 2012 at 4:46 AM, Ben Shum <email address hidden> wrote:

> Going back to James' original bug report, I think we can actually
> confirm that while we were still on 2.0, the located URI's behaved
> strangely for us. For whatever reason, the subfield 9 acted oddly when
> used at the library level vs. the system level. In our system, we have
> a three tier structure (CONS, SYS, BR) and subfield 9 entries at BR
> didn't behave as expected to restrict to only that BR for display while
> subfield 9 entries at SYS did; as far as restricting view of said
> located URIs.
>
> I think this behavior began to change in 2.1+ to match expectations but
> whatever fixes there were never backported fully to 2.0. And since we
> had altered all our located URI's anyways, we didn't get to figuring out
> the actual problem itself.
>
> Will try looking up more details when I get the chance (though 2.0 is
> dead to me!)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/925776
>
> Title:
> located URIs appear in staff client OPAC searches regardless of $9's
>
> Status in Evergreen - Open ILS:
> New
>
> Bug description:
> EG 2.0.10
> Postgres 8.4
> Ubuntu Lucid
>
> The located URIs seem to appear (as greyed out records) regardless of
> scoping within the staff client OPAC (but not the public OPAC).
>
> I can't really think of a use case for this -- if I'm searching with
> my OPAC set to BR1 and "This Branch", I don't want to see 856s with a
> $9 BR2 under that circumstance -- I'd scope out if I wanted to see
> them.
>
> ~James
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/925776/+subscriptions
>

[Mike Rylander (mrylander)]

Dan, I like your idea. As an added bonus, we can simply switch on/off the #staff modifier based on that checkbox instead of based on whether we detect that we're running inside XULRunner. Should be a simple switch.

As for whether located uris are acting properly in this context, that's a sticky question, really. The idea of them is to act almost like copies, but not quite. Consider:

 * scoping on copies should show you records /at or under/ you ranged scope. That is, it's simply broadening the cone through which you're looking
 * scoping on located URIs, however, should show you records that you're /authorized/ to see based on the ranged scope. That is /only when your cone (ranged scope) is small enough to confirm(ish) your authorization/ and not broader. Stated differently, where your ranged scope is contained within the at-or-below list of the OU of the located URI.

So, a consortium-scoped search shouldn't show system or branch scoped located URIs, but a branch scoped search /should/ ... I hope that doesn't muddy the waters...

The tests are (in practice, essentially) the same, but parameters are reversed.

Dan's proposed checkbox (at least, based on my interpretation and implementation proposal) would be a great first step in making this work well for most (if not all) workflows, I think. Having a (maybe staff only) "restrict based on located uri" checkbox might get us all the way there, assuming the logic above is sound, and indeed implemented that way (now or in the future).

--miker

[Ben Shum (bshum)]

Hi Beth,

I'm curious from your specific example about the "Spanish Civil War". In your opac (presumably http://catalog.sage.eou.edu/opac/en-US/skin/default/xml/index.xml) , the only record I could see that was electronic from that search appeared to be an electronic record without a located URI 856 $9 entry. At least at consortium level search for me.

Do you have a specific example you can link to describing the misbehavior of 856 $9 entries in your 2.1 system?

[Beth Longwell (blongwel)]

Ben,

The misbehaving is in the staff client. OPAC appears to be functioning as
it should in regards to hiding located URIs appropriately. Would it help
for you to have staff client access to our production server?

The search example was in the 2.1 staff client, with the search narrowed
down to Lakeview Library (under Lake County Library District).

Beth

On Wed, May 23, 2012 at 11:58 AM, Ben Shum <email address hidden>wrote:

> Hi Beth,
>
> I'm curious from your specific example about the "Spanish Civil War".
> In your opac (presumably http://catalog.sage.eou.edu/opac/en-
> US/skin/default/xml/index.xml) , the only record I could see that was
> electronic from that search appeared to be an electronic record without
> a located URI 856 $9 entry. At least at consortium level search for me.
>
> Do you have a specific example you can link to describing the
> misbehavior of 856 $9 entries in your 2.1 system?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/925776
>
> Title:
> located URIs appear in staff client OPAC searches regardless of $9's
>
> Status in Evergreen - Open ILS:
> New
>
> Bug description:
> EG 2.0.10
> Postgres 8.4
> Ubuntu Lucid
>
> The located URIs seem to appear (as greyed out records) regardless of
> scoping within the staff client OPAC (but not the public OPAC).
>
> I can't really think of a use case for this -- if I'm searching with
> my OPAC set to BR1 and "This Branch", I don't want to see 856s with a
> $9 BR2 under that circumstance -- I'd scope out if I wanted to see
> them.
>
> ~James
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/925776/+subscriptions
>

[Jason Stephenson (jstephenson)]

We seem to have two conversations going on here. Can anyone using tpac in the staff client verify the described behavior?

If not, then we'll set to Won't Fix.

[Beth Longwell (blongwel)]

Jason,

We are using TPAC in 2.2 and can verify the appearance of bibs with located URIs for other libraries when scoped to a specific library in the staff client. The bib summary information displays but the link does not so there is no way to tell that it is online resource until you click on it, only to find out you don't have access to it anyway so it is a frustration to staff who often think it is an empty bib in our system. I can provide screen shots if that would be helpful.

Thanks,

Beth

[Ben Shum (bshum)]

Confirmed for me too, I think. Took me awhile to duplicate the behavior, but I'm also seeing bib records with located URIs show up in searches in the staff client that shouldn't include them for the library I was searching at.

[Jennifer Pringle (jpringle-u)]

Confirmed that this is still a problem with the TPAC in the staff client in 2.4.

[Liam Whalen (whalen-ld)]

I have a fix for this issue that I plan to apply tonight to Sitka's produciton servers.

It may be possible to modify it to return all URI items if using an OU with an ID of 1 by adding a second luri_org_list variable that populates with actor.org_unit_decendants as well as ancestors. However, I am not sure if 1 is an unmutable value for the top level OU, or if there is an easy way to retrieve that value via pgsql.

http://git.sitka.bclibraries.ca/gitweb/?p=sitka/evergreen.git;a=commit;h=222c418d4e3af5077b95f4098d98d1a9ee9ad1c9

Liam

[Liam Whalen (whalen-ld)]

Our sysadmin had a look at my fix to ensure everything was in order, and he notice I was executing a redundant query. This new commit gets rid of that query:

http://git.sitka.bclibraries.ca/gitweb/?p=sitka/evergreen.git;a=commit;h=7878d6d13f4cc97c31caa474d2f4033f5fae4731

I will post this to my evergreen user branch soon.

Liam

[Liam Whalen (whalen-ld)]

One more optimization:

http://git.sitka.bclibraries.ca/gitweb/?p=sitka/evergreen.git;a=commit;h=0b7054687f6448f13aba8b828bf1c614f75d40e5

Liam

[Kathy Lussier (klussier)]

Liam,

Is this branch ready for testing? If so, can you add a pullrequest tag?

[Liam Whalen (whalen-ld)]

The branch has been modified again, and we have been running the new version at Sitka since October 23rd, 2013. These modifications allow searches performed at the top level OU to return all ebooks and searches performed at second level OUs to return all ebooks underneath those OUs.

The second level functionality is probably going to be specific to how the OU tree is used. In the case of Sitka, second level OUs represent library Federations, so it is useful for librarians to see all ebooks underneath a Federation.

I am hesitant to put a pull request on this because this solution will not work for all libraries. I could add library setting that could be checked to include this functionality. If that makes sense, then I will do that. It would be a simple addition, but I want to make sure adding an LSE check within query_parser_fts is worthwhile. I have already added some conditionals to a fairly central query, so I would like to make sure that the effects are not too detrimental on result speeds.

I have pushed my changes to:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/ldw/lp925776_located_uris_removed_from_staff_client_based_on_856_9

This does not contain a file for OpenILS/src/sql/Pg/upgrade/. I looked in there and found a number of files that create query_parser_fts, so I need some guidance on how to apply this change to those files.

As well, I need to figure out if/how a pgTAP test should be made for this function.

Once a decision has been made about including a LSE within query_parser_fts, I have added the appropriate upgrade scripts, and a pgTAP test is created, then I will add a pull request to these changes.

For the time being, if any system needs help applying these changes to their EG instance please contact me.

Liam

[Liam Whalen (whalen-ld)]

Looking over this code today made me notice that the check for is_top_level_ou was not filtering by staff accounts. I have added this code into the working branch, but I have not had time to check it, so please test this before using it.

[Liam Whalen (whalen-ld)]

I am testing these new changes on my dev server, and expect to have them in production soon. My old version of the code returns non-visible objects in OPAC searches scoped to the top level OU.

As for the library setting, I am thinking a string setting would work. In the string, libraries could define a comma separated list of OU levels to perform top down e-book retrieval. So, if your second and fifth level OUs needs to see all e-books available to OU underneath them, then you would assign 2, 5 to the setting.

This would have to be an non-inherited setting because it would be too expensive to search up the OU tree to find a setting if none were set for the search OU. In the case of a setting with no definition (or an invalid definition) the code would assume that no bottom down retrieval is necessary.

Regarding the pgTAP test, I would like to be involved in helping write it, but writing a comprehensive test for query_parser_fts is beyond my ability. I would appreciate any help that could be provided.

Finally, is there documentation that describes how the scripts in the sql/Pg/upgrade/ directory are organized? I am not sure why multiple scripts create the same function and if I need to replace all those functions with my changes or if I need to add a new script with my changes.

[Ben Shum (bshum)]

So, for the upgrade directory, those changes are only applied sequentially for folks upgrading their database over time. The various upgrade script files are later combined to create the version-upgrade script files when releases are cut. Generally, a change to the database involves altering the stock function in the regular files used and then also creating an accompanying XXXX.change_to_be_made.sql upgrade file in the upgrades directory (which is later stamped with the next available number in the sequence by the core committer putting it in for master). That way, new Evergreen installations will just make use of the stock DB files and using the latest version of the SQL vs. existing and running systems that need to be upgraded making use of the small upgrade files in necessary sequential order.

For example, in our case, running a master system, we check for the most recent numbered script in our config.upgrade_log and then we run all the scripts numbered after that. So like if our last one was 0820, then we'd run from 0821 onwards to get "upgraded". When the same function is changed multiple times over the course of many upgrade script files, that's you seeing the iteration of the feature/functionality over time as it evolves with new development and bug fixes.

So, generally speaking, I'd only include actual changed functions and new seed data in upgrade script files. You do not necessarily have to copy other upgrade scripts or replace untouched functions along the way.

I'll leave pgTAP test ideas for someone else who's more confident of that area to explain...

[Liam Whalen (whalen-ld)]

I had a new thought on the library setting. Since it will not be inherited, it can be a boolean setting. Then if an OU needs to return e-books in a top down manner, the setting can be set to true for that OU and left undefined or set false for all others.

[Ben Shum (bshum)]

For note to reviewers, I stumbled a few moments before I realized that Liam's initial working branch is based on rel_2_4 not master.

[Liam Whalen (whalen-ld)]

Sorry about the lack of communication on the branch I was using.

I have made these changes on Sitka's production database.

My next step is to get some input about the effects on speed that adding a check for a library setting might make on searches. I will ask on #evergreen about that.

If there is not too much concern about the speed effects, then I will make the changes to allow libraries to specify which OUs should perform top-down ebook searches.

After that, I think the code will be complete minus any errors people find that need to be corrected. So, I will make the update script as well and then proceed on the pgTAP files.

After the pgTAP files are done, I will make a 2.3 and master version of everything.

Time line is unknown right now.

[Rogan Hamby (rogan-hamby)]

I tested this on current master with the concerto data and everything worked fine with no speed issues. If time allows I will test it with production data as well on a 2.4.4 system. I checked a diff and there is not difference between the 2.4.4 file and the current master.

[Rogan Hamby (rogan-hamby)]

I hit post too quickly. I used concerto but also uploaded a bib ("Private Sector Development in the Third World") with an 856 tag and $9 set to BR3 in SYS2. With the staff client set to BR1 in SYS1 I had the following behavior:

Pre patch - OPAC did not observe the record unless search scope was set to SYS2, staff client set to any search scope saw the record

Post patch - OPAC did not observe the record unless the search scope was set to SYS2 (as before), staff client now saw it only if the search scope was set to SYS2 or CONS (consortium), now it correctly did not display for SYS1

So, it worked as intended.

[Rogan Hamby (rogan-hamby)]

Works correctly on 2.4.4 with production data, no apparent speed issues seen.

[Liam Whalen (whalen-ld)]

Thank you for testing Rogan. I will make the modifications to allow a library setting to specify which OUs will perform topdown searches.

Liam

[Martha Driscoll (mjdriscoll)]

We are starting to see this bug too. We are on 2.4 and started adding $9 to our electronic resources and libraries are seeing the out of scope records appearing in their search scope. It would be great to see this added to core Evergreen. I would be willing to test on our 2.4 production system and 2.5 test system.

[Mike Rylander (mrylander)]

First, Liam, thank you for your work looking at this.

Having investigated the issue, the problem is that we're not rechecking for Located URIs that are outside our scope at the end of the staff-type visibility testing. I've added exactly that check, a scope-inverted query for URIs. This respects all settings (such as the new "acts-as-copy" global flag) and scoping requests (preferred OU, search OU, search depth) without extra effort.

It does not piggy-back filtering of e-resource records on limit-to-available as Liam's does. If consensus is that limit-to-available should filter out e-resource records, we can make that change, but it's my opinion that just because a resource is electronic does not mean it's not available. The record type can be used for that purpose, as well, and if we don't trust the MARC (fair) then "limit to physical" or "exclude online resources" options seems like a less confusing route.

Here's the branch:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp925776_staff-uri-visibility

[Mike Rylander (mrylander)]

Note that backporting to 2.5 will require a different upgrade script.

[Rogan Hamby (rogan-hamby)]

I agree with not piggy backing on limit-to-available. With e-records there
are a different set of circumstances to consider in terms of what
availability means. If we get a 2.5 backport to this I will eagerly test
it.

On Fri, May 2, 2014 at 10:45 AM, Mike Rylander <email address hidden> wrote:

> Note that backporting to 2.5 will require a different upgrade script.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/925776
>
> Title:
> located URIs appear in staff client OPAC searches regardless of $9's
>
> Status in Evergreen - Open ILS:
> Confirmed
> Status in Evergreen 2.5 series:
> New
>
> Bug description:
> EG 2.0.10
> Postgres 8.4
> Ubuntu Lucid
>
> The located URIs seem to appear (as greyed out records) regardless of
> scoping within the staff client OPAC (but not the public OPAC).
>
> I can't really think of a use case for this -- if I'm searching with
> my OPAC set to BR1 and "This Branch", I don't want to see 856s with a
> $9 BR2 under that circumstance -- I'd scope out if I wanted to see
> them.
>
> ~James
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/925776/+subscriptions
>

[Mike Rylander (mrylander)]

Rogan, I plan to make a 2.5 version RSN. I'll update with that.

[Mike Rylander (mrylander)]

I've added a 2.5-ish version here: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp925776_staff-uri-visibility_25

If/when this is merged, 2.5 and 2.6 should get a separate upgrade script numbers so that upgrades to 2.6 from a 2.5 that has this will get the correct version. A stub for the 2.5 upgrade script can be created for 2.6 to fill the hole.

[Rogan Hamby (rogan-hamby)]

I've tested this now on a test box with production data and compared it to production with 2.5.2 on test and 2.5.3 on production using half a dozen records with different org scopes and keyword and title searches. It looks good.

[Martha Driscoll (mjdriscoll)]

I tested this on 2.5 and it looks good. When searching for a title with located URI's I got the following results:

- searching consortia found the record and displayed all links

- searching scoped to a non-owner did not find the record at all

- searching scoped to an owner found the record and displayed only the owner links

- in the client, searching scoped to an owner found the record and displayed only the owner links along with the user's preferred library links

[Brent Mills (brent-5)]

Tested on 2.5.1 with production data and it looks good as well.

Same behavior as mjdriscoll reported. Records are scoping correctly to their org units and according to the record's $9 field

[Ben Shum (bshum)]

Tested this with master and it does look better.

Picked the master version to master and rel_2_6; as for rel_2_5, used the backport branch.

For note, the upgrade script stamps were: 0882 is the 2.6/master version and 0883 was the 2.5 one, though a placeholder for 0883 exists in master for continuity.