Do not update bib source on match-only merge

[Ben Shum]

Evergreen master / 2.8

When bug 957466 was included during 2.8's development, it also added that the bib source would be updated during vandelay. Unfortunately, this created a workflow issue for our catalogers, where the bib source would be updated to whatever was specified during acquisitions loads and even if it was a match-only merge where the record didn't actually change, the bib source would be altered from whatever it was originally to a new one. In our case, this caused complications because of our acquisitions workflow where a new source (like "acq record") overrode the actual bib source in system (like "oclc").

So, while this is a nice feature to prevent bibs with no source from being created, etc. we do not wish to see the source always updated on merges. Either avoid updating source on this type of merge, or add some sort of setting (global flag? YAOUS?) to control how bib sources are updated.

[Ben Shum (bshum)]

In the meantime while we discuss how to proceed for the community at large, as a (terrible) workaround, this code was removed from our vandelay.overlay_bib_record:

        IF v_bib_source IS NOT NULL THEN
            update_fields := ARRAY_APPEND(update_fields, 'source = ' || v_bib_source);
        END IF;

[Kathy Lussier (klussier)]

Here's an idea. How about adding fields to the merge profile that control whether this profile should update the bib source and if it should update the editor and edit date?

It seems like the most logical place to add this setting. The merge profile is what controls which MARC tags are added, replaced, etc. The user setting up the merge profile could then make the determination of whether the MARC changes are extensive enough to justify updating the last-edit and bib source inofrmation.

[Mary Llewellyn (mllewell)]

I think Kathy's idea has merit. Control is what we're looking for in this situation. Sometimes we'd like a source change, sometimes we don't. I definitely don't want my acq libraries matching an OCLC record with a vendor record and changing the source, but it would be nice when we're overlaying vendor records with full records if we could change the source during that load. I'm using SQL queries to do source housekeeping at this point. While we're at it, having control so that the loader can change the TCN during an overlay as it does when we use Z39.50 would make my life complete. But again, only for certain merge profiles for the reason stated about acq loads.

[Rogan Hamby (rogan-hamby)]

I like this idea. There are definitely sources that I don't want to
necessarily overwrite existing sources, especially in acquisitions.

On Wed, Oct 14, 2015 at 4:12 PM, Kathy Lussier <email address hidden>
wrote:

> Here's an idea. How about adding fields to the merge profile that
> control whether this profile should update the bib source and if it
> should update the editor and edit date?
>
> It seems like the most logical place to add this setting. The merge
> profile is what controls which MARC tags are added, replaced, etc. The
> user setting up the merge profile could then make the determination of
> whether the MARC changes are extensive enough to justify updating the
> last-edit and bib source inofrmation.
>
> --
> You received this bug notification because you are subscribed to
> Evergreen.
> Matching subscriptions: evergreenbugs
> https://bugs.launchpad.net/bugs/1447746
>
> Title:
> Do not update bib source on match-only merge
>
> Status in Evergreen:
> New
>
> Bug description:
> Evergreen master / 2.8
>
> When bug 957466 was included during 2.8's development, it also added
> that the bib source would be updated during vandelay. Unfortunately,
> this created a workflow issue for our catalogers, where the bib source
> would be updated to whatever was specified during acquisitions loads
> and even if it was a match-only merge where the record didn't actually
> change, the bib source would be altered from whatever it was
> originally to a new one. In our case, this caused complications
> because of our acquisitions workflow where a new source (like "acq
> record") overrode the actual bib source in system (like "oclc").
>
> So, while this is a nice feature to prevent bibs with no source from
> being created, etc. we do not wish to see the source always updated on
> merges. Either avoid updating source on this type of merge, or add
> some sort of setting (global flag? YAOUS?) to control how bib sources
> are updated.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1447746/+subscriptions
>

[Kathy Lussier (klussier)]

Adding a note that if we use something like a update_bib_source column in vandelay.merge_profile, we would want to set it to True for the stock Full Overlay profile and to False for the stock Match-Only Merge Profile.

When uploading records using a merge profile where this field is set to True, the bib source, last editor, and last-edit date should be updated.

[Dan Pearl (dpearl)]

Kathy: The code right now updates these fields in the vandelay.overlay_bib_record function:

vandelay.queued_bib_record (fields imported_as and import_time)
biblio.record_entry (source, editor and edit_date)

In the last message, you implied that one column in the merge_profile table would be enough to allow or suppress updating of the record_entry fields. Just checking to see if that is still OK.

Also, I am assuming that the update of the queued_bib_record table is OK regardless. Let me know if I'm wrong.

[Kathy Lussier (klussier)]

IMO, yes, one field is all that is required.

The merge_profile identifies which fields (if any) in the MARC record are updated and, therefore, is the best indicator as to whether that MARC record really was edited in the mind of the end users. If the end user considers it edited, then I expect that they would expect the bib source, editor, and edit_date to all be updated.

However, that's just one person's opinion. It would be good to hear from others in case I'm off base.

Also, yes, I would expect the queued_bib_record fields to be updated regardless of which merge profile is used.

[Dan Pearl (dpearl)]

Here is an implementation that has passed some initial tests.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dpearl/LP1447746_import_flag

[Kathy Lussier (klussier)]

Hi Dan,

I've tested this on a clean install, and it looks good so far. I'm going to test an upgraded system shortly. In the meantime, it might be worthwhile to add a test here. At a minimum, we might want a pgTap test that verifies the existence of the new column, but the tests could be more involved than that. See http://irc.evergreen-ils.org/evergreen/2016-05-03#i_245165 where there was further discussion on this.

Kathy

[Dan Pearl (dpearl)]

For now, here is a branch with a pg_tap test for column existence added. It doesn't make a lot of sense to check the data in the column as that is under user control and subject to user whim.

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dpearl/LP1447746_import_flag_A

[Kathy Lussier (klussier)]

Thanks for the pgTap test Dan! The code looks good and works as expected both on a new install and on an upgraded system.

For future reference, in case you have reason to write a live test similar to the one Dyrcona described in that IRC log, we generally write them to work with the Concerto dataset so that we don't need to worry about changes individual sites or users may make to that data.

Merged to master for inclusion in release 2.11.

[Jason Etheridge (phasefx)]

pgTap failure, here:

http://testing.evergreen-ils.org/~live/archive/2016-05/2016-05-13_16:00:02/test.21.html

affects test for bug 957466

[Kathy Lussier (klussier)]

Opening this bug back up again to deal with a fix to the test for bug 957466. I have a branch in the working repo at

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/kmlussier/lp1447746-fix-vandelay-upload-test

This branch:

*fixes the test
*adds a new test to verify that this feature works as expected in either the true or false state
*and renames the test since the functionality from this bug supersedes the functionality that was previously being tested.

The fix only needs to be merged to 2.next since the feature wasn't backported to previous releases.

[Kathy Lussier (klussier)]

Force-pushed an update to the branch. In my rush to get the fix out, I neglected to change the success messages for the SELECT isnt tests. Many thanks to bshum for pointing it out to me.

Also, many thanks to Remington whose initial tests were the basis for the ones that I added.

[Ben Shum (bshum)]

pgtap test runs successfully and seems logical to me. Pushed to master to fix things. Thanks Kathy!