parts ordering enhancement development proposal

[Tim Spindler]

The parts of a bib item currently appear in the list of Monograph Parts ordered by label. This is not especially useful because it is an alphanumeric sort: DISK 10 will appear between DISK 1 and DISK 2, for instance. Evergreen has already built-in the ability to provide a better sort (using the label_sortkey), but it is not fault-free, and doesn’t seem to be uniformly used in the code
To address shortcomings in the sort presentation, Part 1 of this enhancement allows consortium (and possibly library) staff to impose an ordering on the parts in cases where the default ordering is insufficient.

Part 2 provides the ability for consortium staff to eliminate duplicate parts of the improper format with the one of the correct format by providing the ability to merge parts.

Dan Pearl, developer at C/W MARS is currently working on this project.

[Tim Spindler (tspindler-cwmars)]

This is the full specification.

[Kathy Lussier (klussier)]

Hi Tim,

Thanks for posting the full spec!

A question came up at last week's MassLNC conference regarding the merging of parts that I didn't see addressed in the spec. Will this development deal appropriately with holds that have already been placed on the part that is being merged? As an example, if you were merging Disc 1 and Disc 01 into a single Disc 1 label, I would expect all of the existing holds on Disc 01 to be moved to Disc 1.

Thanks!
Kathy

[Dan Pearl (dpearl)]

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dpearl/partord

This has some interference with the merge parts work: LP#1099979

[Ben Shum (bshum)]

Reviewed and it seems to make sense to me. However as I worked on testing this, I found that record summary of copies does not take the part order into consideration. Using the concerto dataset (record 84, for example), I found that the order of copies was based on call number, barcode, etc.

I think this is based on what we see in Open-ILS/src/perlmods/lib/OpenILS/Application/AppUtils.pm (see this commit for an example of a recent change for copy ordering -- http://git.evergreen-ils.org/?p=Evergreen.git;a=commit;h=d21d1dac8f0212764de5a2c3e352ceebc2d63ef5 )

Setting this bug back to incomplete pending further review and work.

[Jason Stephenson (jstephenson)]

Something else I noticed, is that there is an upgrade script in the sql/Pg/ directory. It should be moved to sql/Pg/upgrade/ and have XXXX. added to the front of the filename. This way anyway checking it will know there is an upgrade script to run. It will also guarantee the upgrade script is properly versioned and in the right place when this goes into master.

[Mike Rylander (mrylander)]

I have just a couple comments that I think would help improve the code.

First, I strongly recommend that the sort_order column be made NULLABLE and not get a default. There are at least 2 good reasons for this, one is practical future-proofing, and one is conceptual:

  1) with a NULLABLE column, we can later refine the underlying query such that manually-ordered rows are displayed either first or last, separately from the sort order, by using the NULLS FIRST/LAST modifier to the ORDER BY component. This is important because there are two classes of sorted things -- manually sorted and automatically sorted -- and they should be segregated based on that fact, not flipped first/last based on ASC/DESC sort.
  2) NULL better models the intent of the new feature -- we are saying "humans are not overriding the natural order, so the human-supplied order is undefined". That is quite different, conceptually, from "humans are not overriding the natural order, so the natural order assumes this row comes first".

Second, Ben identified the exact the spot in the code where copy ordering is applied for the TPAC (and serials display in the SC -- see the code comment above the sub containing the commit-linked code). As it turns out, bmp is already available in the FROM clause of that query, so adding the same order_by elements seen here* in between the call number label sort key and the copy number element should do just what we want. That is, sort first aou.name, then on call number, then human-part-order, then natural part order, then copy number, then barcode.

Jason's cleanup of the upgrade script is also important, thanks for pointing that out.

Overall, this patch has good code economy and does the right things in the right places. Solid work, Dan.

* http://git.evergreen-ils.org/?p=working/Evergreen.git;a=blob;f=Open-ILS/src/perlmods/lib/OpenILS/Application/Search/Biblio.pm;h=69fc2aec7dbdb2c30a343aa11bf8b9e4117760c9;hb=b5ee64f7001a9437639915b831ca08c8cd33c4fb#l2640

[Dan Pearl (dpearl)]

Thank you for your comments, Mike, Jason and Ben. I am OK will all of them except for the NULL of the sort value.

I understand the visceral appeal of having a two-level approach where a human-part-order takes precedence over the natural-part-order. My issue is that the human factors would be a bit awkward for the users of the facility. By default, the nulls will be sorted last, after the explicit values.

If one wanted to put a part named "Index" at the end of all the volumes (v. 1, v. 2, etc.), then one would need to give explicit numbers to all volumes, and give the index a higher number. So this use case devolves into my original implementation, after a bit more work by the user.

Another use case. Say I want to put "Introduction to the Collection" before the numbered volumes. This one is easier: I'd only have to give the Introduction a number -- any number -- and it will display before the (null) volumes. This is equivalent to the work required in my original implementation, where the user would put, say, -1 as the sort number, which would be displayed before the default 0's.

So while the null design has aesthetic appeal, I think it would be harder to use.

[Mike Rylander (mrylander)]

Dan,

I'll defer to staff input on the point of NULLability. If it's the normally the case that the goal, when ordering parts, is to order them globally if a human order is ever used, then NOT NULL (as you have it) makes the most sense. If it's normally a matter of picking one and pushing it to the top (or bottom) then I think NULLABLE would be better, IMO.

Later, we'll probably want to consider a drag-and-drop UI instead of giving the user direct access to the positional value. I wouldn't call it a blocker, in any way, for this to get in, but there is other code in admin interfaces to base that on.

Thanks!

[Dan Pearl (dpearl)]

Thank you, Mike.

I agree about drag-and-drop being nicer, and that would be a reasonable followup.

By "staff input", who are you referring to, and how can we proceed to get closure on this?

[Tim Spindler (tspindler-cwmars)]

Speaking for our staff at C/W MARS, we would number everything in our usage and would not just do one or two. However, in the future it would be great to have a drag and drop UI. Dan and I talked about this at one point and there were some issues in this first go around.

[Mike Rylander (mrylander)]

I was referring to any library staff that want to weigh in ... like Tim! So, with that, I'll withdraw any and all objections.

Thanks, Dan. Nice work.

[Kathy Lussier (klussier)]

I would like to weigh in too. Could somebody frame the question in end user speaker?

[Kathy Lussier (klussier)]

That is, end user speak. Sorry for the noise.

[Tim Spindler (tspindler-cwmars)]

I think Dan in his statement above where he writes

"If one wanted to put a part named "Index" at the end of all the volumes (v. 1, v. 2, etc.), then one would need to give explicit numbers to all volumes, and give the index a higher number. So this use case devolves into my original implementation, after a bit more work by the user.

Another use case. Say I want to put "Introduction to the Collection" before the numbered volumes. This one is easier: I'd only have to give the Introduction a number -- any number -- and it will display before the (null) volumes. This is equivalent to the work required in my original implementation, where the user would put, say, -1 as the sort number, which would be displayed before the default 0's."

You could have say 3 parts

v.1
v.2
v.3
Index

you then have a column in the table specifying sort order.

part sort order
v.1 1
v.2 2
v.3 3
Index 4

if no order is applied in Dan's code you have the following where sort label would take over

part sort order
v.1 0
v.2 0
v.3 0
Index 0

[Dan Pearl (dpearl)]

We're OK with the NON NULL version. I have made the other changes suggested by the reviewers. Thanks!

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/dpearl/partord
has been updated.

[Kathy Lussier (klussier)]

Hi Dan,

I took a look at this code today. Overall, it looks good. I just have a couple of questions.

1. The detailed view of the search results page does not appear to be honoring the new sort order. I would expect parts to be sorting here the same way that they do on the bib record page. See

https://mlnc1.mvlcstaff.org/eg/opac/results?query=sports%20illustrated;qtype=keyword;locg=1;detail_record_view=1

in comparison to

https://mlnc1.mvlcstaff.org/eg/opac/record/238

2. The xul-based holds interfaces of the staff client can display a holds column that is sortable. Should we expect these interfaces to honor the imposed sort order? In my testing, they are still sorting alphabetically. This is a general question for users.

Thanks Dan!

Kathy

[Dan Pearl (dpearl)]

Kathy: The links in your comment don't produce anything useful, but I might be able to intuit what is going on.

1. The sort hierarchy is 1. Org Unit, 2. Call Number label sortkey 3. monograph part label sortkey 4. copy number 5. Barcode It looks like some libraries put what should be a part designation as part of the call number. I don't blame them, of course, but because of the hierarchy, it takes precedence. Personally, I think it may be more useful to flip #2 and #3 in the sort hierarchy, but the behavior is how it was specified.

2. I agree that this is a general question. My rudimentary understanding of the XUL interface is that when it provides column sorting, it sorts on the data within the column, and not by some independent sort key. While an extension in this area may be useful, I suspect that the implementation would be complicated. Also of concern is whether a non-XUL staff client would make work in this area moot.

[Kathy Lussier (klussier)]

Hi Dan,

I'm sorry it took so long to follow-up again. I have re-added the parts code on the MassLNC server to demonstrate what I'm talking about. It's a testing server, so it gets reloaded on a regular basis and will need to be available for other code on bug squashing day. Therefore, I have also taken some screenshots in case you don't have a chance to look at it between now and then.

1. To be clear, the record I am using is one in which all of the items are using parts. It is not a case where the part designtation is part of the call number.

On the bib record, the item information is displaying in the sort order I specified. See http://mlnc3.mvlcstaff.org/eg/opac/record/239 or http://www.screencast.com/t/7AUsb5b5LMQ4

On the search results screen, the item information seems to be displaying in a reverse sort order. Seehttp://mlnc3.mvlcstaff.org/eg/opac/results?query=sports%20illustrated;qtype=title;locg=1;detail_record_view=1 or http://www.screencast.com/t/ENIV65tX

2. Well, call numbers are sorting by the sortkey in the staff client, so it's not unprecedented. Also, I suspect there may be several sites remaining on the xul client for at least another year, so I don't think it's entirely moot. However, I also don't think this is a show stopper. However, I do think users will expect to see consistency between the search results and bib record page.

Thanks Dan!

[Kathy Lussier (klussier)]

I think Dan Scott's frequent comments about the differences between sorting on the search results page and the record details page have finally sunk in with me. In thinking about this further, Dan P., I think the reason we're seeing this discrepency on the search results page is because sorting there is handled through in-db unapi calls. I don't see any changes to 990.schema.unapi.sql in the branch, which might explain why the sorting didn't carry over to the search results page.