Bib records that are both deleted and empty show in staff search

Bug #1773832 reported by Jason Stephenson on 2018-05-28
34
This bug affects 6 people
Affects Status Importance Assigned to Milestone
Evergreen
Medium
Unassigned
3.0
Medium
Unassigned

Bug Description

Evergreen Version 3.0.8
OpenSRF Version 3.0.1
PostgreSQL Version: 9.5.13
Host O/S: Ubuntu 16.04.4

We just upgraded from 2.12.8 to 3.0.8 and noticed that bib records that are both empty (i.e. have no call numbers/volumes) and deleted show up in staff search.

It is our understanding that empty bibs that are not deleted should show up in staff search, but that deleted bibs should not show up in staff search, even if empty, unless the #deleted flag is added to the search.

Revision history for this message
Jason Stephenson (jstephenson) wrote :

I should clarify, this is true for both the web staff client and XUL client.

These bibs do NOT show up in patron OPAC search, only when searching via the staff clients.

Revision history for this message
Kathy Lussier (klussier) wrote :

I tried this in the web client on a couple of test VMs: one was a clean master install and the other was an upgraded database.

I added a record and didn't add any holdings. I performed a search and was able to find the test record, as expected.

I deleted the record. When I performed the exact same search, I was able to retrieve the record. However, if I performed a different search, I got no results, leading me to believe that the first search was relying on cached data. In my testing, I wasn't able to replicate this bug except in those instances.

I think there was an issue in the early 3.0 days where deleted records were retrieved in staff client searches, but I can't find the bug number.

Revision history for this message
Kathy Lussier (klussier) wrote :

I may be seeing different behavior because I was adding the records after the new search visibility code was added to the system. It may only be an issue for records that were there before the upgrade. There were other search visibility issues that only occurred on upgraded records, not on ones that were added after the upgrade.

Revision history for this message
Jason Stephenson (jstephenson) wrote :

Kathy, if I change the search, to title and author instead of keyword, I still get the deleted records in the results.

You may be right that it has to do with records that existed before the upgrade. In that case, it seems that something is missing from the upgrade instructions. Perhaps the bib vis attr update needs to be run on deleted records, too?

Revision history for this message
Jason Stephenson (jstephenson) wrote :

I tried updating the vis_attr_vector on deleted bibs using the steps from the upgrade. It does not help. The deleted and empty bibs still show up in search. I have tried different variations from before to avoid cached results. I also stopped services in between.

Revision history for this message
Elaine Hardy (ehardy) wrote :

We discovered this last week. The deleted records show up in or result set as if they are blank records (see attached screenshots. When you click on one to display, you just get a white screen. In order to see the deleted record, we have to click on a previous record and then page through to the deleted one.

Changed in evergreen:
status: New → Confirmed
Revision history for this message
Elaine Hardy (ehardy) wrote :
Revision history for this message
Elaine Hardy (ehardy) wrote :

If I do keyword or title search, the deleted records show up in the staff client (both XUL and web client 3.0.2).

If I do a TCN search in the web client, I do not retrieve the record and get a message Bib Record Not Found: {TCN}, A record ID search retrieves the deleted record.

In the XUL client, when I search for a deleted record with the TCN, a dialog box appears telling me that record is deleted and asking show the record anyway? This is preferred behavior since catalogers might have the TCN of a deleted record and need to see the record in case it was deleted in error and should be undeleted. They are unlikely to have the database ID (Our TCNs are based on the OCLC TCN, primarily)

In the XUL client, a database/record ID search retrieves the deleted record.

Kathy Lussier (klussier) on 2018-05-29
tags: added: opacvisibility
tags: added: searchvisibility
removed: opacvisibility
Revision history for this message
Michele Morgan (mmorgan) wrote :

I am not able to reproduce this issue on a 3.0.7 system. Neither a newly added empty record which is subsequently deleted, nor an existing empty deleted record are showing up in staff searches. I tested in both the xul and web clients.

Revision history for this message
Pamela Smith (pamela-smith) wrote :

We are on 3.06 and long deleted records are appearing in both the staff & web clients. Records added & deleted after the upgrade are appearing also. Title, author,keyword & record ID searches all retrieve delete records.

Revision history for this message
Kathy Lussier (klussier) wrote :

Jason's initial report noted that this is happening with empty deleted bibs. For those who are confirming this bug, is it also only happening with deleted records or is it happening with other records too?

Revision history for this message
Kathy Lussier (klussier) wrote :

Sorry, my question should have been:

Is it also only happening with *empty* records?

Revision history for this message
Janet Schrader (jschrader) wrote :

I'm not sure what you mean by *empty* records. Undeleted records with no items can show in the staff client depending on how long the network wants to keep them. We keep them for 6 months from the date added for OCLC records and 1 month for brief records libraries create.
CWMARS has the same experience Pamela describes. The records are visible in both xul and web client. For us, they show in results list and link to cover images and Novelist content. We do not have the blank records Elaine describes.
I also noticed that when a record is deleted this is indicated, in both xul and web, in the record summary area by red highlight or the word "deleted" respectively and in the bib display by a red outlined message. If I undelete a record these indicators go away as intended. If I subsequently delete the record a second time only the record summary area indicates the record is deleted. The bib display does not have the red bordered message.

Revision history for this message
Janet Schrader (jschrader) wrote :

Forgot to say we are on 3.0.8.

Revision history for this message
Elaine Hardy (ehardy) wrote :

I have only seen deleted records and not empty records (bib records brought into the systems where holdings are not yet attached).

We have far more deleted records than empty records so that might be a factor. Cataloging workflows generally mean empty records get processed within a week or so, although some do get left behind.

Revision history for this message
Kathy Lussier (klussier) wrote :

Sorry to be unclear. What I meant to say was that Jason originally reported that it was happening to records that were both deleted AND empty (no holdings). I have no concerns about undeleted records showing up in staff client searches.

However, from your recent comments, it sounds like you're all reporting that many different types of deleted records are showing up in a search, not just the empty ones.

Revision history for this message
Janet Schrader (jschrader) wrote :

Yes, for CWMARS it is both deleted and empty. I tried to delete a record with holdings but am not able to do so. For that the system is working correctly.
As for the delete/undelete/delete I described above, the red bordered message in the bib record display does finally appear when a record is deleted a second time, so there is a delay there.

Revision history for this message
Elaine Hardy (ehardy) wrote :

Empty vols should show up in a staff search, so I would expect them to be in the list. But deleted records shouldn't. And it appears that the deleted records displaying are not recent deletes necessarily

Revision history for this message
Mike Rylander (mrylander) wrote :

Here's a branch that forces an explicit check on the deleted flag of biblio.record_entry:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1773832-hide-deleted-bibs

From the commit message:

There are cases where we avoid checking the deleted flag of bibs as an optimization that avoids a JOIN in the search query. However, for some shapes of real-world data, particularly in upgraded instances, deleted records can slip through. This commit adds an explicit check for the flag to all search queries.

tags: added: pullrequest
Revision history for this message
Dan Pearl (dpearl) wrote :

I tested it and it worked well.

I have tested this code and consent to signing off on it with my name,
Dan Pearl, and my email address, <email address hidden> .

tags: added: signedoff
Changed in evergreen:
importance: Undecided → Medium
milestone: none → 3.1.3
Revision history for this message
Jason Stephenson (jstephenson) wrote :

I tested the branch, too, and it is working for me, so I added Dan's signoff and pushed to master, rel_3_1, and rel_3_0.

Thanks, everyone!

Changed in evergreen:
status: Confirmed → Fix Committed
Changed in evergreen:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers