Search: Identifier Field Virtual Index Mappings not working

[Michele Morgan]

Evergreen Master

Stock Evergreen is configured with virtual index mappings (config.metabib_field_virtual_map) that map identifier fields to the keyword All Searchable Fields (id 45 in config.metabib_field)

When performing a keyword search for a term that exists in metabib.identifier_field_entry, but does not exist in metabib.keyword_field_entry, the record is not retrieved.

An example from the concerto data is record id 246, Ready player one E-audio.

The identifier scn (field 29) has the following index vector in metabib.identifier_field_entry:

'749998216':4A 'erc0000004936':2A 'ocolc':3A 'tlc':1A

An identifier scn search on terms 749998216 or erc0000004936 will retrieve the record, but a keyword search on either term does not retrieve the record.

As another example, the identifier isbn (field 18) for record 246 has the following index vector:

'0307913163':2A '9780307913166':1A

The 10 digit isbn is not present in the marc, so is not present in the keyword index vector. The 10 digit isbn exists in metabib.identifier_field_entry because of the ISBN 10/13 conversion normalizer.

An identifier isbn search on 0307913163 retrieves the record, but a keyword search on 0307913163 does not.

This was brought to our attention by library staff that noticed the discrepancy when searching isbns, but the scn may be a clearer example as there is no normalizer involved.

Some background:

Virtual index mappings were introduced in bug 1744385.

Here's a link to the documentation:

https://docs.evergreen-ils.org/docs/reorg/3.3/staff_client_admin/_virtual_index_definitions.html

[Jessica Woolford (jwoolford)]

We ran into this when attempting to add a virtual index map for Genre to All Subjects. As Michelle noted, if a term appeared in the metabib.identifier_field entry, but not in metabib.subject_field_entry, it did not appear in the search.

[Mike Rylander (mrylander)]

Michele, if you change the weight of the virtual mapping (which is the weight of the source field in context of a search against the sync) to be different from the baseline weight of the sync field, and then restart services just in case, do the searches work as you expect? I think there may be a too-clever-by-half optimization used to cut down on the join count that's getting in the way.

[Michele Morgan (mmorgan)]

Hi Mike,

Thanks for looking at this! Your suggestion did work on my concerto test system.

After increasing the weight on the virtual mapping for ISBN, and restarting services, my searches worked as expected. Keyword searches on 749998216 and 0307913163 (examples from the original but report) retrieved the record as I would expect given the virtual mappings.

Here's what I did on my concerto system:

Baseline weight of the sync field:

SELECT weight
FROM config.metabib_field
where name = 'blob'

weight: 1

ISBN virtual mapping id and weight:

SELECT map.id, map.weight
FROM config.metabib_field_virtual_map map
JOIN config.metabib_field field ON map.real = field.id
WHERE field.name = 'isbn'

id: 13
weight: 1

Confirm that a keyword search on 0307913163 gets no hits

Change the weight on the virtual map to 2:

UPDATE config.metabib_field_virtual_map
SET weight = 2
WHERE id = 13

Restart services

Perform a keyword search on 0307913163 using cache killer, for example:

0307913163 -lmlsflk;ijfs

The search retrieves the record as expected.

This is great news, and likely gives us a workaround, but can the optimization be tweaked to allow the virtual indexes to work without adjusting weights?