Search highlighting not working for words with diacritics

[Linda Jansova]

When words such as "veda" (ASCII-only) are entered into TPAC search box, Evergreen 3.1.0 correctly highlights the search term in search results.

However, when words such as "sborník" (containing a non-ASCII character "í") are entered, the word fails to appear highlighted in search results.

Example:
https://eg-test.osvobozena-knihovna.cz/eg/opac/results?query=sborn%C3%ADk&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0

[Eva Cerninakova (ece)]

I have tested this in EG 3.1.1 and in EG master. In both cases the problem still exist.
Words containing non-ASCII characters are not highlighted in search results

If the phrase is being searched containing both word without non-ASCII character and word with non-ASCII characters, only the "non-ASCII part" of the search query highlighted in search results (see the image).

[Mike Rylander (mrylander)]

This is a sided effect of long-standing (and, until highlighting) user-invisible search code.
 Specifically, it's because we force the use of search_normalize() against both indexed and search input terms, which strips diacritics. This is addressable, but unfortunately fairly non-trivial.

[Michele Morgan (mmorgan)]

This is a problem for catalogs of all languages. Words and names with diacritics are not uncommon in all catalogs. Here are a few examples of words/names containing diacritics from our English language catalog:

Authors, artists, etc.:
Charlotte Brontë
Carlos Castañeda
Federico Garćia Lorca
François Rabelais
Mary GrandPré
Paul Cézanne
Camille Saint-Saëns

Names and loan words from French and other languages:
Esmé
Renée
exposé
fin de siècle
Les Misérables

I'm attaching screenshots of searches of some of the above terms to illustrate the issue.

[Michele Morgan (mmorgan)]

Catalog search for Charlotte Brontë

[Michele Morgan (mmorgan)]

Catalog search for Paul Cézanne

[Michele Morgan (mmorgan)]

Catalog search for exposé

[Michele Morgan (mmorgan)]

Catalog search for Les Misérables

[Eva Cerninakova (ece)]

Still the isssue in Bootstrap OPAC, see the attachment

[Linda Jansova (skolkova-s)]

It is still the case in the most current Evergreen; confirmed using a Mobius test server available at https://bugsquash2.mobiusconsortium.org/eg/opac/home. Please see the attached screenshot.

[Christine Morgan (cmorgan-z)]

Confirming that this is still an issue. Testing on https://terran-main.gapines.org/ during Bug Squashing Week.

[Linda Jansova (skolkova-s)]

I have just checked that the issue still persits, e.g.:

https://bugsquash2.mobiusconsortium.org/eg/opac/results?query=si%C3%A8cle&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=

(The query string used was siècle.)

[Linda Jansova (skolkova-s)]

It is still a relevant issue in the currently tested Evergreen; see for example Blake's bugsquash server and search results for siècle: https://bugsquash.mobiusconsortium.org/eg/opac/results?query=si%C3%A8cle&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=&show_highlight=0&show_highlight=1.

[Linda Jansova (skolkova-s)]

I confirm it is still relevant; I have just tried to search for siècle on Blake's bugsquash server }like I did in August 2024): https://bugsquash.mobiusconsortium.org/eg/opac/results?query=si%C3%A8cle&qtype=keyword&fi%3Asearch_format=&locg=1&detail_record_view=0&sort=&show_highlight=0&show_highlight=1.