Preserve specification in Merge/Overlay Profiles not working as expected

[Dale Rigney]

Tested in Evergreen 3.3 and Evergreen 3.4

When using the MARC Batch Import interface the Preserve Specification field in the Merge profile is not being applied on merged records.

In our test we used a Merge profile that would preserve tag 490. Our marc record in the database had a tag 490$a and the incoming record did not have any 490 tags. We expected the 490$a tag to remain after the merge. The record was merged but tag 490$a was removed.

The Preserve Specification field does work in Evergreen 3.2 (which uses the old style vandelay interface). Evergreen 3.3 and 3.4 uses the new Angular vandelay interface.

[Andrea Neiman (aneiman)]

Confirmed in 3.4

[Angela Kilsdonk (akilsdonk)]

Confirmed in 3.3 as well

[Mike Rylander (mrylander)]

Thanks, all, for surfacing this.

Was an item import attempted in the run where the preserve rule failed?

[Andrea Neiman (aneiman)]

In my tests, I only attempted bib import with no associated items.

[Rosie Le Faive (rlefaive)]

This is a pretty darn important feature of Vandelay to anyone who's using it.

If the Angular interface was released without knowledge that this was failing in practice...

Has the community moved on from using Vandelay? Should we be using other tools?

[Evergreen Bug Maintenance (bugmaster)]

Noting that this appears to have been a regression caused by the patch for bug 712490. That change was made to one of the underlying stored procedures; it doesn't represent Evergreen moving away from Vandelay.

[Galen Charlton (gmc)]

Whoops, that last comment was by me.

[Mike Rylander (mrylander)]

I don't have a fix for this right now, but I have identified the issue in the code. Commit 23293c7f reworked how the replace rules were processed to allow, say, subfield 0 to remain at the end of the field. However, it now requires that the "target" record (the new record in the case of a preserve rule) have the named field in order for the value from the "source" (old record) to be moved over.

[Mike Rylander (mrylander)]

I have a branch available here:

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1878984-vandelay-preserve

From the commit message:

When processing a Preserve specification, vandelay.replace_field fails to account for the situation where the target record does not have any instances of the field to be preserved. This commit copies fields from the source to the target when there are no like target fields and the source fields are not restricted by a match requirement.

It additionally adds multi-subfield match requirement capabilities to the add_field and replace_field functions, as implemented in the branch submitted for LP#1901597 that does the same for strip_field.

[Jennifer Weston (jweston)]

Tested three different scenarios during Feedback Fest -- all worked as expected using merge profile with preserve specifications.

Scenario #1
490 on existing bib in db; no 490 on new record; 490 field was preserved during merge

Scenario #2
no 490 on existing bib in db; 490 on new record; 490 field was not loaded from new record during merge and, therefore, the state of the existing record was preserved

Scenario #3
245a on existing bib is good and should be preserved; 245a on new record is not correct and should not be loaded; good 245a was preserved during merge

I have tested this code and consent to signing off on it with my name, Jennifer Weston, and my email address, <email address hidden>

[Janet Schrader (jschrader)]

I tested this on the Equinox Test Server 2. I used the existing merge/overlay profile Keep 490 that has preserve 490 and 901. In case anyone wants to look, I loaded the file "CW Series statements", 5 records each with a 490 field. Then I loaded "CW Test Keep 490 merge profile", the same five records with no 490 fields and each with a new 500 field. The TCNs are 268-272.

I can confirm that the series statements were preserved.

But there is a quirk I wasn't expecting. In the queue you can click in the row to open the record. This opens the record as it appears in the queue, not necessarily as it appears in the database. If you have overlayed a record you only see the updated record if you click on the number in the "imported as" column. In other words there are three different versions of the record depending on how you open it.

In the current Vandelay, there is a column to see the record as it existed in the queue you loaded before it got into the database. If there are matches the link to the match shows the most updated version of the record. It's more obvious a merge/overlay profile worked.

Janet

[Jason Boyer (jboyer)]

Hi Mike, I was looking at this today and noticed that the functions in the upgrade script and 012.schema.vandelay.sql don't match. Could you sync those up?