Parts Management Slow with Large Numbers of Parts

[Jason Boyer]

Eg 2.7.2
Osrf 2.3
Pg 9.1
Ubuntu 12.04

We have some serials records with up to 200 parts, and loading these records in the Configure Monograph Parts editor takes nearly 40 seconds. I've verified that the database returns results quickly via the logs. Using Chrome I profiled the JS running in the staff client/browser when rendering this page and dojo seems to be doing a lot of slow calls attempting to determine the proper sizing for the table that the list is built in. The fast fix would seem to be pagination since other conify interfaces work fine (usually with a max count of 10-15) but hopefully this can more fully addressed when we're able to move away from Dojo altogether. I can add some profiling data if it helps (and if there's a decent format for it, or screenshots.)

[Kathy Lussier (klussier)]

Thanks Jason! It looks like this is the same issue as was reported in bug 1193468. I'm marking this one as a duplicate.

[Mike Rylander (mrylander)]

Hrm... I don't think this is actually a dupe. The other refers to the tpac record detail page, and this is about the parts editor, iiuc.

That said, while I think we should work to improve the performance of any interface, the use case described (using parts as a stand-in for serials) is not the intended use of the interface. IOW, the parts UI is intentionally optimised for a certain data shape and size, and the serials UIs are optimised for another.

[Kathy Lussier (klussier)]

Sorry, Mike, you're right. I didn't read the description closely enough.

In our case, I think we understand that parts were never intended for serials data, however, the fact remains that we will always have libraries that choose not to use the serials module. In fact, in the two MassLNC consortia that use serials, I believe it is just a small percentage of libraries that choose to use serials. The other consortium doesn't use serials at all. It comes down to the fact that creating distributions, prediction patterns, issuances, etc. is a lot of overhead for a small public library that has subscriptions to a few popular magazines that they keep for about a year before discarding them.

One MassLNC consortium is using parts for serials because it allows those libraries to quickly add copies through cataloging and to also make them available for holds. An added benefit is that the holds can be filled by any library in the consortium using the same part, whereas, IIRC, issuance holds are similar to volume-level holds.

I think there is development that can be done to handle serials information for libraries that will never dive into the full serials module, but, in the meantime, this is the workaround that has worked for at least one consortium in Massachusetts.

[Mike Rylander (mrylander)]

Kathy,

Totally understand. Just to be perfectly clear, I agree that if we can make it faster for all use cases, we should.

Regarding issuance holds being like volume holds, that really depends on how you use the serials system. If you maintain one MFHD record and allow all libraries to hang their subscriptions off of it, issuance holds are more like part holds. Actually, their broader than that -- they're closer to title holds, as you /can't/ have two issuances with the same label on a single MFHD. However, if every location has their own MFHD with their own (single, active) subscription, then they behave more like volume holds.

In the case of a small library that is just keeping a year or maybe two of a magazine, would that normally amount to 200+ parts? I would think that would be much less. To whit...

Jason, is the record in question something like a newspaper with daily issues, of which your library is keeping a goodly back catalog?

TIA.

[Jason Boyer (jboyer)]

It's a weekly (People) that some locations have back to 2013. It's slightly inflated in that some of them come from merged records and some duplicate parts need to be merged away, but roughly 3 years worth of weeklies is over 150 so it won't likely come down significantly.

The issue appears to be dojo specifically. I profiled it building the list in Chrome and I'll attach a screenshot. For anyone following along that hasn't messed with Chrome's JS Profiler, the Y axis is time (from 0 ms to about 1.8 mins ) and the X axis is the stack depth, so the taller the graph the more calls are being made until something "finishes" and returns back up to a high enough level to repeat everything for the next part (when the graph goes to 0 no JS is running), and the color bars at the bottom show the functions called between the two vertical bars in the top graph.

You can see the peaks where new rows are inserted start to separate at around 16 seconds* on this run and they're quite far apart near the end around the 1.7 min mark (on a 3GHz Xeon E5-1607 with 16GB of RAM). dojo.declare.updateRowCount start around 6 ms for the first row and is taking 1.34 seconds between rows at the highlighted point on the graph. I don't think there's anything that can be done in Evergreen except either make that interface page (makes merges harder) or wait until dojo is out of this interface (which isn't in the cards right away since this is an interface that can be run as-is in the web client).

If there's any other data that would help to have let me know. If anyone is curious about the database, it only took 535ms to return these parts.

* this is not 16 seconds from the initial return of data, I had to login first which pushed things back 7-8 seconds.