Web Client: Patron Search sorted by last name not working as expected

[Terran McCanna]

In 3.0.0...

We're seeing unexpected results when sorting by last name in the patron search results screen.

See attached screenshots:
On demo.evergreen.org I modified some of the Smiths in order to test and searched by smith. As you see in the screenshot, Smith-Smith appeared in an odd place, and S'mith didn't appear at all.

On our PINES test server, I searched by dale, and the D'Alessandros appeared in different spots in the results.

[Terran McCanna (tmccanna)]

Tested again in 3.0.1, still not sorting correctly.

[Terran McCanna (tmccanna)]

Sorting solely by DOB (either thru column config or by clicking on column header) doesn't work either. See example.

[Andrea Neiman (aneiman)]

I was definitely seeing some wonkiness (but slightly different wonkiness) in demo.evergreencatalog.com yesterday, but I failed to screenshot it and now I'm having trouble getting into that system.

On a separate 3.0.0 test system I was not seeing any of the issues with Patron Search Results -- First Name, Last Name, Middle Name, DOB, Create Date all sort up & down completely as expected.

HOWEVER, then I changed a patron's last name from Davis to D'avis, and now I see the following sort:

D'avis
Daniels
Davenport
Davis
(etc)

Where I would expect sort to ignore the apostrophe.

Not sure if that is related. I can open a separate bug if it's another issue.

[Terran McCanna (tmccanna)]

I'm still seeing significant problems with last name sorting. See examples when I put "dal" as the last name - It does push most of the names beginning with d'al to the top, but then within those names it's not sorting correctly.

Further down the list in the "dalber" section it has "dalbert" before "dalberg" etc.

It's almost like it's sorting in clumps so that things are sort of grouped together but not actually in order.

[Terran McCanna (tmccanna)]

To expand on this, first name and middle name sorting are off as well. The attached examples all had the same last name and were sorted by clicking on the column headers.

[Joan Kranich (jkranich)]

Release 3.0.3

When I search by last name only, the first names are in alphabetical order and then the alphabetizing starts over again.

In one test the first 16 results are in order and then line 17 starts again with first names beginning with the letter 'a'.
In most tests the first 25 results are in order and then the sorting starts over again.
I have the rows set to 50.

I am not clicking the column headers to sort just retrieving results.

[Jason Stephenson (jstephenson)]

I'm setting the importance to High because it has been identified to me as critical to fix this.

[Bill Erickson (berick)]

Has anyone been able to reproduce this on the test/concerto data set? It may not be possible, given the limited amount of data, but very helpful if possible.

Barring that, it would be helpful to know if the sorting is off in the patron search database query or if the UI is getting them jumbled. Hint, search for "unaccent_and_squash" in the PG logs.

[Terran McCanna (tmccanna)]

Hi Bill - the first example and attachment at the top of this thread was in master with Concerto data, but I did change a few of the Concerto patron names to make it easier to test.

[Bill Erickson (berick)]

Thanks, Terran.

I am unable to reproduce :( I have 25 "Smith" patrons, a "S'mith", a "Smith-Smith", and a "Smithfield". They reliably display in the expected order S'mith => Smith => Smith-Smith => Smithfield.

Are any column "Sort Priority" values applied to columns in the Manage Columns interface?

[Terran McCanna (tmccanna)]

I can no longer reproduce it on Concerto, so that's good.

We are still seeing it with our full data set on 3.0.2, and don't have a 3.1 test server set up with our full data yet, so I'm not sure if something has been fixed between the two versions or if it only occurs once the data set becomes too large. It's almost as if it's loading a handful of records at a time, sorting those, then loading the next set and sorting those. I'm attaching another set of screenshots where I did a last name search, then sorted the results by first name (highlighting shows the places where there was a glitch in sorting.)

I confirmed that no priority values were set in the Manage Columns interface, and also tested on multiple browsers and machines to make sure there wasn't a stored preference or cache complicating things.

Once we have a 3.1 test server with our full data set on it, I'll test further.

[Jennifer Pringle (jpringle-u)]

We're seeing this on our live 3.1 server.

[Chris Sharp (chrissharp123)]

More data here.

I did searches on "Sharp" and "Dale" in the PINES database, and the sorting is wrong in the UI as described, but when running the query directly via psql, the sorting is correct.

[Bill Erickson (berick)]

Chris, just to confirm, you said in IRC that running the patron search API via srfsh returned results out of order?

[Bill Erickson (berick)]

Fix pushed:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/berick/lp1724029-patron-search-api-ordering

There appears to be a problem with the OpenSRF max_chunk_size API decorator. For reasons not yet clear to me, it results in responses getting returned in the wrong order. I have replaced the use of max_chunk_size with the preferred max_bundle_count. This does not suffer from the sorting issues and its meaning is more clear in the code.

Note I reviewed the code and the only other cases of max_chunk_size I found were setting the value to 0, which effectively disables it, so we shouldn't have any more of these lurking.

[Michele Morgan (mmorgan)]

Tested this patch on a 3.1.4 server with production data. Prior to loading the patch, searches on the last name "dale" when sorted by last name consistently showed sequences like:

Dalelio
Dalessandro
Dalelio
Dalelio

and

Daley
Daley
Dalessio
Daley
Daley

Sorting by first name consistently showed sequences like:

Angela
Austin
Bernadette
Bob
Alex
Amanda

and

Craig
Crystal
Brian
Brian
Brittany
Brittany
Brooke
Cailin
Caitlin

After applying the fix, all fields are now sorting as they should. My signoff branch is at:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/mmorgan/lp1724029-patron-search-api-ordering-signoff

[Bill Erickson (berick)]

Thanks, Michele! I've merged to 3.0 and up.