Preserve specification in Merge/Overlay Profiles not working as expected

Bug #1878984 reported by Dale Rigney
This bug affects 4 people
Affects Status Importance Assigned to Milestone

Bug Description

Tested in Evergreen 3.3 and Evergreen 3.4

When using the MARC Batch Import interface the Preserve Specification field in the Merge profile is not being applied on merged records.

In our test we used a Merge profile that would preserve tag 490. Our marc record in the database had a tag 490$a and the incoming record did not have any 490 tags. We expected the 490$a tag to remain after the merge. The record was merged but tag 490$a was removed.

The Preserve Specification field does work in Evergreen 3.2 (which uses the old style vandelay interface). Evergreen 3.3 and 3.4 uses the new Angular vandelay interface.

Revision history for this message
Andrea Neiman (aneiman) wrote :

Confirmed in 3.4

Changed in evergreen:
status: New → Confirmed
importance: Undecided → High
Andrea Neiman (aneiman)
tags: added: van
tags: added: cataloging vandelay
removed: van
Revision history for this message
Angela Kilsdonk (akilsdonk) wrote :

Confirmed in 3.3 as well

Revision history for this message
Mike Rylander (mrylander) wrote :

Thanks, all, for surfacing this.

Was an item import attempted in the run where the preserve rule failed?

Revision history for this message
Andrea Neiman (aneiman) wrote :

In my tests, I only attempted bib import with no associated items.

Revision history for this message
Rosie Le Faive (rlefaive) wrote :

This is a pretty darn important feature of Vandelay to anyone who's using it.

If the Angular interface was released without knowledge that this was failing in practice...

Has the community moved on from using Vandelay? Should we be using other tools?

Revision history for this message
Evergreen Bug Maintenance (bugmaster) wrote :

Noting that this appears to have been a regression caused by the patch for bug 712490. That change was made to one of the underlying stored procedures; it doesn't represent Evergreen moving away from Vandelay.

tags: added: regression
Revision history for this message
Galen Charlton (gmc) wrote :

Whoops, that last comment was by me.

Revision history for this message
Mike Rylander (mrylander) wrote :

I don't have a fix for this right now, but I have identified the issue in the code. Commit 23293c7f reworked how the replace rules were processed to allow, say, subfield 0 to remain at the end of the field. However, it now requires that the "target" record (the new record in the case of a preserve rule) have the named field in order for the value from the "source" (old record) to be moved over.

Revision history for this message
Mike Rylander (mrylander) wrote :

I have a branch available here:;a=shortlog;h=refs/heads/user/miker/lp-1878984-vandelay-preserve

From the commit message:

When processing a Preserve specification, vandelay.replace_field fails to account for the situation where the target record does not have any instances of the field to be preserved. This commit copies fields from the source to the target when there are no like target fields and the source fields are not restricted by a match requirement.

It additionally adds multi-subfield match requirement capabilities to the add_field and replace_field functions, as implemented in the branch submitted for LP#1901597 that does the same for strip_field.

tags: added: pullrequest
Andrea Neiman (aneiman)
Changed in evergreen:
assignee: nobody → Andrea Neiman (aneiman)
Andrea Neiman (aneiman)
Changed in evergreen:
assignee: Andrea Neiman (aneiman) → nobody
Changed in evergreen:
assignee: nobody → Jennifer Weston (jweston)
Revision history for this message
Jennifer Weston (jweston) wrote :

Tested three different scenarios during Feedback Fest -- all worked as expected using merge profile with preserve specifications.

Scenario #1
490 on existing bib in db; no 490 on new record; 490 field was preserved during merge

Scenario #2
no 490 on existing bib in db; 490 on new record; 490 field was not loaded from new record during merge and, therefore, the state of the existing record was preserved

Scenario #3
245a on existing bib is good and should be preserved; 245a on new record is not correct and should not be loaded; good 245a was preserved during merge

I have tested this code and consent to signing off on it with my name, Jennifer Weston, and my email address, <email address hidden>

tags: added: signedoff
Changed in evergreen:
assignee: Jennifer Weston (jweston) → nobody
Revision history for this message
Janet Schrader (jschrader) wrote :

I tested this on the Equinox Test Server 2. I used the existing merge/overlay profile Keep 490 that has preserve 490 and 901. In case anyone wants to look, I loaded the file "CW Series statements", 5 records each with a 490 field. Then I loaded "CW Test Keep 490 merge profile", the same five records with no 490 fields and each with a new 500 field. The TCNs are 268-272.

I can confirm that the series statements were preserved.

But there is a quirk I wasn't expecting. In the queue you can click in the row to open the record. This opens the record as it appears in the queue, not necessarily as it appears in the database. If you have overlayed a record you only see the updated record if you click on the number in the "imported as" column. In other words there are three different versions of the record depending on how you open it.

In the current Vandelay, there is a column to see the record as it existed in the queue you loaded before it got into the database. If there are matches the link to the match shows the most updated version of the record. It's more obvious a merge/overlay profile worked.


Revision history for this message
Jason Boyer (jboyer) wrote :

Hi Mike, I was looking at this today and noticed that the functions in the upgrade script and 012.schema.vandelay.sql don't match. Could you sync those up?

Michele Morgan (mmorgan)
tags: added: needsrepatch
removed: pullrequest signedoff
tags: added: needswork
removed: needsrepatch
Elaine Hardy (ehardy)
tags: added: cat-batchedit
removed: cataloging vandelay
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.