MARC 852 field columns misaligned on z3950 export

[Dmagick]

Evergreen version presented 2.0.5, verified still present in 2.1.1

The 852 subfields produced in a typical export from the staff client of a MARC record are coded in the export script in the file marc_export as follows:

MARC::Field->new(

852, '4', '',

a => $location,

b => $orgs{$cn->owning_lib}->shortname,

b => $orgs{$cp->circ_lib}->shortname,

c => $shelves{$cp->location}->name,

j => $cn->label,

($cp->circ_modifier ? ( g => $cp->circ_modifier ) : ()),

p => $cp->barcode,

($cp->price ? ( y => $dollarsign.$cp->price ) : ()),

($cp->copy_number ? ( t => $cp->copy_number ) : ()),

($cp->ref eq 't' ? ( x => 'reference' ) : ()),

($cp->holdable eq 'f' ? ( x => 'unholdable' ) : ()),

($cp->circulate eq 'f' ? ( x => 'noncirculating' ) : ()),

($cp->opac_visible eq 'f' ? ( x => 'hidden' ) : ()),

)

However, when a record is derived via z3950, these columns are aligned differently via the code presented in SuperCat.pm:

                           MARC::Field->new(
                                '852', '4', '',
                                a => $cp->{'location'},
                                b => $bib_holdings->{$cn}->{'owning_lib'},
                                c => $cn,
                                d => $cp->{'circlib'},
                                g => $cp->{'barcode'},
                                n => $cp->{'status'},
                            )

Part of the resulting issue is an inability to work with the $j, $y and $p fields as presented by Evergreen when 3rd parties or other retrieving entities require this price, status, and identification information for their own products and services that they would otherwise expect having examined typical exports.

[Josh Stompro (u-launchpad-stompro-org)]

Hello, just confirming that this is still an issue in 2023, checked 3.11 and master around 7/26/2023.

It looks like the bib bucket export also uses different codes than the marc_export script as well.

https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/WWW/Exporter.pm;hb=HEAD

Uses subfield z for status instead of s, and doesn't have call number prefix/suffix subfields at all. Also sets subfield x to nonreference,holdable,circulating,visible vs marc_export that only does the opposites of those values.

Also it sets the MARC Location Code from LOC (subfield a) to "gaaagpl" as the default value vs the marc_export script that excludes that subfield unless a flag is set to set the value.

Where else should I check for 852 generation that may also be in a different format?

Our new state wide ILL system may be using both exported dumps from marc_export and checks via z39.50 so having two wildly different formats will be just awkward to explain.

Changing the z39.50/sru format may break our current ILL integration since it also does availability checks via z39.50. So this will probably need to be well documented in a major release.

It seems to me like the marc_export format more closely follows the MARC standard for 852.
https://www.loc.gov/marc/holdings/hd852.html

Does anyone know of any reason to not have one standard for 852 generation for all evergreen holding exports?

Josh

[Jason Stephenson (jstephenson)]

Josh,

I don't know of any reason to have more than 1 format for 852 generation.

I reimplemented marc_export several years ago to make it faster. When I did so, I followed the LoC standard rather closely, although I think the original marc_export was fairly close to the standard to begin with.

I think a decent way to deal with this would be to factor out the export code from marc_export into a Perl library and use that on the back end wherever we need to export MARC. If you look, you'll see that marc_export is already divided up into internal modules. That was done partly with the though of splitting it up in the future and partly just as a way to organize the code.

As regards subfield a, I've considered opening a bug to add the national library code to the org unit settings or somewhere else configurable in the database. I'll wager many organizations don't know what there's is.

Jason

[Mike Rylander (mrylander)]

I wonder if we should use the YAOUS cat.marc_control_number_identifier wherever we need the national library code? that's used as the authority record source identifier in the $0 when we link to an authority in the bib, and in the 003 and 035 when updating the bib control number fields.

Side note: gaaagpl is, IIRC, the lowercase version of PINES' OCLC or LoC (or both?) library code.

[Josh Stompro (u-launchpad-stompro-org)]

Ha, looks like the OAI-PMH uses another mapping for the 852 subfields.

https://evergreen-ils.org/documentation/release/RELEASE_NOTES_3_9.html#_oai_pmh_data_provider_support

But those mappings are easily changed in the opensrf.xml config.
https://git.evergreen-ils.org/?p=Evergreen.git;a=blob;f=Open-ILS/examples/opensrf.xml.example;h=05d945fdebd728b3561eb90ac02a4096dffb9948;hb=HEAD#l859

Josh

[Josh Stompro (u-launchpad-stompro-org)]

I'm going to take a stab at updating the z39.50 holdings format so it more closely matches the marc_export format. I think creating a new perl module is outside my comfort zone though, so I'm just going to update what is there.

For the OAI-PMH responder I have a working branch at bug 2030667 that does the same thing. I really like the ability to customize the output via a config file there. That may also be something to emulate elsewhere. I liked being able to add things like a concatenated full call number, parts and active date.

Josh

[Josh Stompro (u-launchpad-stompro-org)]

Working branch for testing at : user/stompro/lp927764_z3950_supercat_852_format_testparts

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/stompro/lp927764_z3950_supercat_852_format_testparts

This branch supports parts. But currently how unapi.bre() handles adding in monographic parts results in duplicating all the holdings data, which can almost double the xml size and more than doubles the execution time.

I'll look at adding a different includes directive like 'bmp-apc' for unapi.acp to indicate that we only want the monographic_parts fleshed out for copies.

So this branch is mainly for testing. I'll create a new branch that doesn't include the parts next.

This branch Produces the following output (Bunch of extra subfield J's to exercise those optional options).
datafield tag="852" ind1="4" ind2=" ">
<subfield code="b">MOORHEAD</subfield> -- owning lib
<subfield code="b">MOORHEAD</subfield> -- circ lib
<subfield code="c">Main</subfield> -- Shelving Location
<subfield code="k">MINN COLL</subfield> -- Call Number Prefix
<subfield code="j">977.6 Bjo</subfield> -- Call Number
<subfield code="j">MINN COLL 977.6 Bjo</subfield> -- Full Call Number
<subfield code="j">MINN COLL 977.6 Bjo ( Part: 4 )</subfield> -- Full Call Number + Part
<subfield code="j">4</subfield> -- Part
<subfield code="g">Reference</subfield> --Circ Mod
<subfield code="p">33500002677227</subfield> -- Barcode
<subfield code="s">Lib Use Only</subfield> -- Status
<subfield code="y">30.00</subfield> -- Price
<subfield code="e">1</subfield> -- Copy Number
<subfield code="z">reference</subfield> -- Reference Flag
<subfield code="z">unholdable</subfield> -- Holdable Flag
<subfield code="z">noncirculating</subfield> -- Circulating Flag
</datafield>

This effects all use of SRU with holdings option, so it isn't just Z39.50.

Initial Testing Notes:

SRU request like this can be used to test without using Z39.50.
https://<server>/opac/extras/sru/CONS/holdings?version=1.1&operation=searchRetrieve&query=eg.author%3DBj%C3%B6rnson&startRecord=1&maximumRecords=100

[Josh Stompro (u-launchpad-stompro-org)]

Working branch at user/stompro/lp927764_z3950_supercat_852_format

https://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/stompro/lp927764_z3950_supercat_852_format

This branch tries to match the marc_export format as close as possible.

I'll get release notes and documentation updated also.

Josh

[Terran McCanna (tmccanna)]

Adding pullrequest tag to make sure it gets testing attention

[Josh Stompro (u-launchpad-stompro-org)]

Force pushed documentation and release notes.

I added a short page for Z39.50 that mostly refers people to the evergreen wiki page at https://wiki.evergreen-ils.org/doku.php?id=evergreen-admin:sru_and_z39.50#setting_up_z3950_server_support_in_evergreen

I added a page for SRU that also refers to that page, but documents the 852 format also, along with some example requests to help people get started without having to read the SRU standard docs.

I think it is ready for a pull request now.
JOsh

[Jane Sandberg (sandbergja)]

Thanks for the patch, Josh -- it works well for me, and the documentation additions are very welcome.

My only suggestion would be to remove the parts lines and other commented-out lines -- I don't know if their meaning will be clear after a few years. My preference would be instead for a more descriptive comment at the beginning of the block detailing what isn't included in the 852 and why. But I don't consider that a blocker; I'd also be happy to sign off on the branch as it is.

I agree with your comment in #1 that this is a breaking change. I'd like to target this for 4.0, but I'm not sure the best way to do so.

[Josh Stompro (u-launchpad-stompro-org)]

Oops, I'm not sure why I left those extra j subfields in there... I think I meant to remove them before the pull request and missed it.

I've removed the pullrequest.

I'll clean it up so the default 852 matches marc_export, which was my intention. I would really like to copy the OAI-PMH setup that allows for setting the 852 format in opensrf.xml, that would make the change easier, if it was easy to to back to the old format after the upgrade. I'm not sure if I'll have time to work on that any time soon though.

Josh