Quick Export Feature for Vandelay Queues

[Dan Wells]

When we import records into Vandelay, sometimes some of the records are not what we expected, so they get rejected. These records often need to be loaded using a different set of options, so this link makes it convenient to get a new file of just the rejects.

This could be expanded to other sorts of queue exports, but keeping it simple for now.

[Dan Wells (dbw2)]

Branch here:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1548147_vandelay_quick_export

working/user/dbwells/lp1548147_vandelay_quick_export

Basic test:
- Import via Vandelay a file you know will have at least one rejected record (e.g. an exact match overlay with a totally new record id)
- Go to your import queue
- Click the new 'Export Non-Imported Records'; save resulting file

[Mike Rylander (mrylander)]

Dan, had you considered extending the existing export mechanism to accept a vandelay queue id in addition to the various record sources available currently? That mod_perl handler is already optimized to handle a large number of record, and supports various output formats. That would also keep maintenance of record export code in one place, rather than having various version floating in different parts of the code.

[Dan Wells (dbw2)]

I can't recall if I tried that, but if I did, I'm happy to reconsider it. This was just something I did very quickly some time ago which I thought might be useful to share, so I threw it out there. I'll take a look and report back.

[Dan Wells (dbw2)]

Dug a little deeper, and the reason we went this way is that the exporter code currently assumes your ultimate goal is to pull marc from the *.entry_record tables, and the data being exported here never makes it past the vandelay tables, making it quite different in shape and size. Not a big deal (I basically have the new code ready), but enough to give me pause to wonder if we actually want this mixed in. Thoughts?

[Mike Rylander (mrylander)]

Thanks for digging deeper!

If we leave the holdings flag off this round, I think the structural
change would just be to construct the cstore call a little
differently, based on type. The shape of the record is same enough
(the relevant field is called 'marc') that it should work with out
trouble. We'd just need a new block toward the top of handler() that
uses a new cgi param (say, 'bibqueueid') to look at
vandelay.bib_queue. Or, we could expose the parent table and cover
bib and authority queues in one go.

I'm not too afear'd of those changes, personally.
--
Mike Rylander
 | President
 | Equinox Software, Inc. / The Open Source Experts
 | phone: 1-877-OPEN-ILS (673-6457)
 | email: <email address hidden>
 | web: http://www.esilibrary.com

On Tue, Feb 23, 2016 at 4:59 PM, Dan Wells <email address hidden> wrote:
> Dug a little deeper, and the reason we went this way is that the
> exporter code currently assumes your ultimate goal is to pull marc from
> the *.entry_record tables, and the data being exported here never makes
> it past the vandelay tables, making it quite different in shape and
> size. Not a big deal (I basically have the new code ready), but enough
> to give me pause to wonder if we actually want this mixed in. Thoughts?
>
> --
> You received this bug notification because you are a member of Evergreen
> Bug Wranglers, which is subscribed to Evergreen.
> https://bugs.launchpad.net/bugs/1548147
>
> Title:
> Quick Export Feature for Vandelay Queues
>
> Status in Evergreen:
> New
>
> Bug description:
> When we import records into Vandelay, sometimes some of the records
> are not what we expected, so they get rejected. These records often
> need to be loaded using a different set of options, so this link makes
> it convenient to get a new file of just the rejects.
>
> This could be expanded to other sorts of queue exports, but keeping it
> simple for now.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/evergreen/+bug/1548147/+subscriptions

[Dan Wells (dbw2)]

Thanks, Mike. Okay, I'll run with it and see if anything else trips things up, but I expect it'll be alright.

[Dan Wells (dbw2)]

Hello again. Totally reworked branch now here:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1548147_vandelay_quick_export_v2

working/user/dbwells/lp1548147_vandelay_quick_export_v2

[Galen Charlton (gmc)]

Following discussion in #evergreen, I've pushed this to master along with a couple follow-ups to change the export format to MARC21 and to add a test plan and release notes. Thanks, Dan!