We should expand the effect of the fix for 1308090 and remove more ISBD punctuation

[Mike Rylander]

In bug 1308090 Dan Pearl created a function to strip some dangling ISBD punctuation from author-ish fields. We need to improve and expand that to protect some trailing punctuation and to cover some more ISBD punctuation such as dangling "/" and ":" on title-ish fields.

Branch coming soon...

[Mike Rylander (mrylander)]

Branch available at: https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1864507-more-trailing-punct-cleanup

[Jeff Davis (jdavis-sitka)]

I can confirm that the updated trim_trailing_punctuation function works as advertised. I wonder if we should add ";" to the list of trailing characters to be trimmed?

Does anyone have a good test case for replicating the original problem? It seems like we might need a MARC-based config.metabib_field browse field for that; the standard ones are based on the MODS representation of the record, which appears not to contain trailing ISBD punctuation. For example, MARC 245$a = "The fellowship of the ring /" translates into MODS32 titleBrowse = "The fellowship of the ring" and the latter is the basis for the title browse entry.

[Mike Rylander (mrylander)]

I need better gmail filters -- I missed you're update, Jeff, sorry!

If there's consensus, I have no problem adding ";" to the list of trailing punctuation and "=" looks like it might likewise show up based on the LoC docs for MARC 245.

[Jane Sandberg (sandbergja)]

+1 to removing ";" and "="

[Mike Rylander (mrylander)]

Hi folks,

I've force-pushed an update to the above branch that adds equal and semicolon to the list of trailing punctuation we remove now.

[Jennifer Weston (jweston)]

Tested on https://tiffany-master.gapines.org/eg/staff

Loaded the following records as examples of each trailing punctuation to be preserved:
TCN 256
245 10$aOlivia counts =

TCN 257
245 10$aGuidebook to Zen and the art of motorcycle maintenance /

TCN 258
245 00$aProceedings at symposium;

TCN 259
245 00$aAction painting :

Punctuation is successfully preserved upon import.

[Jennifer Weston (jweston)]

Tested during Feedback Fest on https://tiffany-master.gapines.org/eg/staff

Noting this bug adds the following punctuation to the list to be preserved in title-ish fields / : ; =

Loaded the following records as examples of each trailing punctuation to be preserved:

TCN 254
245 10 ‡aGoodnight moon 123 :

TCN 255
245 10 ‡aGoodnight moon = ‡bPw zoo hli /

TCN 256
245 00 ‡aFiddler on the roof; ‡b South Pacific; The sound of music

Punctuation is successfully preserved upon import.

I have tested this code and consent to signing off on it with my name, Jennifer Weston, and my email address, <email address hidden>

[Jane Sandberg (sandbergja)]

Merged for inclusion in 3.10. Thanks, Mike and Jennifer!

I added some release notes based on my understanding of the effects of this patch -- would you have a moment to check those, Mike?

[Galen Charlton (gmc)]

Pushed fix to seed data and schema update scripts; syntax error would have broken the schema update and left some indexes without the new normalizer in a new database.