acq and cataloging loader broken in 2.6-rc

[Ben Shum]

Evergreen master / 2.6-rc

We have a problem with the loader in both acquisitions and cataloging. The following is a short list of steps attempted by Mary Llewellyn who was testing loading records on our system:

Uploaded small file (3 bibs/items)
Create PO, Add to Selection list, new queue
Record Match Set: Matchpoint; Merge Profile: Match only Merge
  import non-matching records, merge on best match
Processed the line items within a minute
   6 lineitems processed, 3 ACQ copies processed
Queue name disappeared after the lineitem numbers appeared
Checked acq queue list and name of queue is not there.
Load appears to keep running and running
Checking the selection list shows the bibs have not yet linked to the catalog

Set up a cataloging load
small file (3 bibs)
Record match set: Matchpoint; Merge profile: overlay bibs
Merge on best match
queue doesn't disappear, shows on list of bib queues
the load keeps running and doesn't get past the Uploading.... Processing...0 step.

Asking in IRC ( http://irc.evergreen-ils.org/evergreen/2014-04-08#i_85694 ), we got confirmation that match-only merge as a choice in loading seems to result in long delay and failure to proceed.

Investigation underway, reporting bug first and noting that this should be a blocker against releasing 2.6.0 if acq/cataloging is so severely impacted.

[Kathy Lussier (klussier)]

I had no problem uploading records in acquisitions when using a merge profile set to preserve the 099 and 690. It was a little slow, but, since I don't load records on a regular basis, I can't say it's any slower than usual.

However, when I use a profile to replace the 901c (match-only merge) I experience similar problems to what Ben reported. No lineitems processed and the queue wasn't created.

I was using a master system that was updated on 4/2/14.

[Dan Wells (dbw2)]

I am not sure how it fits based on the feedback given so far, but this seems likely at face value to be another case of metabib.record_attr no longer being usable (speed-wise) for WHERE clauses (similar to the Z39.50 search bug).

If this is in fact the problem, I think it should be a fairly simple matter to substitute metabib.record_attr_flat directly in the queries, especially since Vandelay is building the queries directly.

[Dan Wells (dbw2)]

Rather then going view->view->data, here is a quick-fix function change which just goes view->data:

http://pastebin.com/CtujPy6z

Mike mentioned he plans to cook up a version which goes straight to the data (for even better performance), but I am curious what the results would be with this updated function, since at least we might discover if we're on the right track.

[Dan Wells (dbw2)]

Ack, was too hasty, had some extra pieces in there. This *might* be better:

http://pastebin.com/SsYRziws

That said, Mike already has his branch done, so we should probably redirect our attention there:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/collab/miker/1304559_and_1271661_vandelay_fixes

[Ben Shum (bshum)]

We applied the upgrade scripts from the vandelay_fixes_upgrades branch and the result was still long running acq/cataloging loads that never completed.

Next we tried Dan Wells' approach noted in the paste for #4 and that seems to have allowed a load to process in a more timely manner. Will follow up on getting that into a working branch and pushed somewhere.

[Dan Wells (dbw2)]

I am surprised Mike's closer-to-metal solution didn't fix this issue. It probably deserves some more eyes, since it should be superior, but in the meantime, here is a branch for my quick fix:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dbwells/lp1304559_record_import_timeout

[Ben Shum (bshum)]

Pushed dbwells fix since it gets us moving again. Will test further, maybe we can still use Mike's approach for 2.7 or future bug fix in a later ticket.