Wishlist: Search & display infrastructure improvements

[Andrea Neiman]

Equinox has been contracted by MassLNC to develop several improvements to Evergreen's search & display infrastructure.

Improvements include:
--Creation & use of Virtual Index Definitions, which will allow an administrator to control weighting and field inclusion in the keyword index. This includes a new interface to manage Virtual Field Data Providers.
--Increased used of Metabib Display Fields, with the eventual goal of replacing the Metabib Virtual Record class in a future release.
--The creation of a Search Term Highlighting feature which will allow a toggled display of highlighted search terms in search results lists, metarecord results lists, and within record details.
--Re-implementation of the Administration interface MARC Search/Facet Fields in Angular JS.

Full functional specifications can be seen here:
https://yeti.esilibrary.com/dev/public/techspecs/search-and-display.pdf

A branch is forthcoming.

Also, noting that I have marked bug 1479113 as a duplicate since the search highlighting it calls for is included in this development.

[Mike Rylander (mrylander)]

Here find a branch implementing Virtual Index Definitions, Search Field Highlighting, and by necessity, increased use of Metabib Display Fields: http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1744385-search-highlighting-virtual-index-defs

[Galen Charlton (gmc)]

I've created a new collab branch for my testing of it: collab/gmcharlt/lp1744385-testing

Note that at the moment it includes a follow-up patch fixing a syntax error that broke creation of new databases.

[Mike Rylander (mrylander)]

I've pushed a new branch including some testing-identified improvements, incorporating Galen's fix above (I could have sworn I had that already, but thanks for catching that, Galen) and rebased against master for hopefully-simpler merge:

http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/miker/lp-1744385-search-highlighting-virtual-index-defs-rebased-fixes

[Mike Rylander (mrylander)]

NOTE: There's one additional improvement that I'd like to do here, replacing single-record MVR creation with Display Field data, but IMO merge does not need to wait for that.

[Mike Rylander (mrylander)]

The branch has been updated with various display and performance improvements, with a few small tweaks to come for the seed data.

[Bill Erickson (berick)]

Started eyeballing and light testing to come up to speed. Initial questions/comments:

* Looks like we lost some sign-off's from Galen's branch. Maybe they weren't official yet?

* Do we still need the Dojo version of the admin page?
Open-ILS/src/templates/conify/global/config/metabib_field_virtual_map.tt2

* Release notes are pending, i assume, though I see one of the commit messages essentially are the release notes.

* use constant DEBUG_TIMING => 1; In EGCatLoader.pm :)

==

* SQL upgraded cleanly
* TPAC searches are showing highlighted results (neat!)

[Mike Rylander (mrylander)]

Bill,

 * I squashed a tiny fix from Galen into the main body of the code ... that's where his signoff went missing. I can force-push it back.

 * We do for the Xul client, I think. No?

 * Yeah, that big commit was going to be the release notes, along with a "remember to reingest!" note.

 * Heh, I'll get that now...

Thanks!

[Kathy Lussier (klussier)]

I've added my signoff branch here:
http://git.evergreen-ils.org/?p=working/Evergreen.git;a=shortlog;h=refs/heads/user/kmlussier/lp-1744385-search-highlighting-virtual-index-defs-rebased-fixes

I also turned Mike's earlier commit message into a release notes entry and made some edits/additions to that release notes entry.

We tested this branch against a fairly large set of test data. The highlighting works well, and we're getting good search results, both in terms of relevance and performance. We're looking forward to seeing this improvement in Evergreen!

[Dan Wells (dbw2)]

This branch was quite an effort, so big kudos to everyone involved, especially Mike, Kathy, and Bill. I nearly broke my pen signing off on this thing!

One thing I noticed in final review which I didn't see mentioned yet, the way highlighting is turned on and off via different data paths leads to some display oddities. For example, in the Concerto set, do a search for 'gaiman', go in a record, then turn highlighting on/off. When highlighting is 'on', the page title is just the title, but when it is off, it reverts back to the complete 245 field. Obviously this isn't particular to these records, but is likely to happen more often than not for most records. I, for one, would be happy to finally jettison the pre-formatted silliness that is MARC field 245 (well, most of MARC, unfortunately), but to have it tied to the highlight state it a bit odd. Not immediately sure on the best direction there, so I'll open another bug for further discussion. Thanks again, all!

[Jeff Godin (jgodin)]

The bug mentioned in Dan's comment 9 above is bug 1752434