Assign stat cats during Vandelay import/overlay of items

[Remington Steed]

When importing or overlaying items through the Vandelay MARC batch import process, there is no way to retrieve statistical categories (stat cats) from the MARC record and apply them to the items in Evergreen.

This could be accomplished by expanding the Holdings Import Profiles table (vandelay.import_item_attr_definition) to include a column for stat cat data, which could be parsed and applied to items during the import process.

Branch is coming soon.

[Dan Wells (dbw2)]

Initial branch of three commits is here:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1379815_vandelay_stat_cat_import

user/dbwells/lp1379815_vandelay_stat_cat_import

[Remington Steed (rjs7)]

I'm removing the pullrequest tag because further testing has revealed a bug. New branch will be posted soon.

[Dan Wells (dbw2)]

Fixed branch force-pushed to same location:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1379815_vandelay_stat_cat_import

user/dbwells/lp1379815_vandelay_stat_cat_import

[Kathy Lussier (klussier)]

HI Dan and Remington,

I just tested this and, overall, it looks good. However, I noticed a couple of things.

The system will import the stat cat even if it belongs to an OU that doesn't own or isn't a child of the OU that owns the stat cat. For example, I created a stat cat for BR1. I imported a record with the stat cat that also had an owning/circulation library of BR2. The stat cat was successfully applied to the record. I know with copy locations, the item import will fail if you try to use a copy location that shouldn't be used by the circulation/owning OU.

Should we be able to import multiple stat cats or is that beyond the scope of this project? A majority of our libraries have at least two stat cats required for their copies, so I know it would be useful for them to import multiple stat cats.

Thanks for your work on this! I know our catalogers will be happy when they can start importing stat cats with their items.

Kathy

[Dan Wells (dbw2)]

Thanks for testing, Kathy. We'll looking into the org scoping issue you mention. Also, yes, you can import multiple categories, by separating with a double-pipe (||), so something like:

CAT 1|VALUE 1||CAT 2|VALUE 2

I know Remington is working on a DB test for this feature, so perhaps he'll find time for release notes as well. It might be mostly a matter of pulling some of this information out of the commit messages.

[Remington Steed (rjs7)]

I added release notes and a pgTAP test. Here is my updated branch:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/rsteed/lp1379815_vandelay_stat_cat_import

[Dan Wells (dbw2)]

Kathy pointed out via IRC that we never addressed the ability of this code to be too liberal in applying stat cats in cases where the category doesn't exist for the given org unit. We will look into the possibility of adding a guard rail to protect against this.

[Dan Wells (dbw2)]

Okay, we did some testing, and from what I am seeing, stat cats aren't being limited in current interfaces by org unit. I created a stat cat owned by BR1, and was able to apply it normally to copies with both owning lib and circ lib of BR2. This isn't what I expected, but I can imagine use cases for this behavior.

I also wondered whether there might be controls in place at the user permission level to limit editing of stat cats applied by other libraries. After checking into it, I am not seeing any permission checking specific to stat cat editing. Basically, if you are allowed to update the copy, it appears that you can modify any stat cats for any library on that copy.

I might be missing something, but at this point, I think the import code models properly how stat cats are treated in other interfaces.

Upon reviewing the code, we did see some opportunities for better overall error handling. Please expect an updated branch soon.

[Remington Steed (rjs7)]

New commit pushed to the same branch for better error handling:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/rsteed/lp1379815_vandelay_stat_cat_import

[Ben Shum (bshum)]

Hmm, what's oils_xpath_tag_to_table ? Cause it's referenced in the changes for the vandelay function, but I can't find it anywhere in our current schemas.

So I got errors like this showing up in my logs where my system can't find that function:

open-ils.cstore 2015-08-18 21:05:40 [ERR :54272:oils_sql.c:2486:14399364235426638] open-ils.cstore ERROR inserting vandelay::queued_bib_record object using query
LINE 2: FROM oils_xpath_tag_to_table( (SELECT mar...
                                ^
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
QUERY: SELECT *
                  FROM oils_xpath_tag_to_table( (SELECT marc FROM vandelay.queued_bib_record WHERE id = import_id), attr_def.tag, xpaths)
                            AS t( ol TEXT, clib TEXT, cn TEXT, cnum TEXT, cs TEXT, cl TEXT, circ TEXT,
                                  dep TEXT, dep_amount TEXT, r TEXT, hold TEXT, pr TEXT, bc TEXT, circ_mod TEXT,
                                  circ_as TEXT, amessage TEXT, note TEXT, pnote TEXT, internal_id TEXT,
                                  stat_cat_data TEXT, opac_vis TEXT )
CONTEXT: PL/pgSQL function vandelay.ingest_items(bigint,bigint) line 195 at FOR over SELECT rows

Is something missing from this branch or am I not having the right functions for some reason? I couldn't find oils_xpath_tag_to_table anywhere in the stock Evergreen code either...

Marking incomplete pending further details on this.

[Dan Wells (dbw2)]

The code for that function is in the first two commits for this bug on Remington's branch. They may have been overlooked since they had a different author than the rest (me instead of Remington).

[Ben Shum (bshum)]

Thanks Dan for pointing out my mistake there. We applied all the commits this time around and tests were successful.

Comfortable merging this to master for 2.9.

Thanks guys! This feature will be very helpful for us.