Elasticsearch choking on non-ASCII characters

[Aaron Wells]

In 15.10 I've added code to "quarantine" records that Elasticsearch won't index. That is, if Elasticsearch errors out while processing a batch of records, then I re-try each record individually. And if it errors out while processing one of those individual records, I mark the record as quarantined, and keep it in the search_elasticsearch_queue table.

I've backported that to one of our large 15.04 sites, and since then I've taken a look at the data in the records that have caused Elasticsearch to choke. They all contain non-ASCII characters, i.e. Unicode characters. These can be as simple as "e with an accent over it", all the way up to exotic ones like emoji and the Unicode snowman.

I was not able to replicate this when testing on my local machine, but it is certainly in place on our production servers, and bugs such as Bug 1408577 make me think it's probably also present on some other servers as well.

[Aaron Wells (u-aaronw)]

For testing purposes, here are a few sample words (in page titles, artefact titles, and user names) that have caused Elasticsearch to choke:

João
Jiménez
Māori

It's not clear from our situation whether the problem lies in our Elasticsearch setup, or in Mahara's code. I think it may be something peculiar to our server setup because I haven't been able to replicate the problem on my local machine.

[Robert Lyon (robertl-9)]

I found a couple of links to do with ASCII folding token filter, which may help

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/analysis-asciifolding-tokenfilter.html

http://stackoverflow.com/questions/28584780/ignore-ascii-characters-on-elasticsearch

[Kristina Hoeppner (kris-hoeppner)]

Similar report at bug #1408577

[Kristina Hoeppner (kris-hoeppner)]

We'll look at this issue when upgrading Elasticsearch to a newer version. Apparently, Elasticsearch can have a few problems with certain languages.

[Robert Lyon (robertl-9)]

I notice that on my local machine Mahara 17.04+ saves

João Jiménez Māori

in the title fields as 'João Jiménez Māori'
and in the description fields as 'João Jiménez Māori'

But on cluster machines in 16.10 it saves

in the title fields as 'Jo<C3><A3>o Jim<C3><A9>nez M<C4><81>ori'
in the description fields as 'Jo&atilde;o Jim&eacute;nez M<C4><81>ori'

If I do a

 SELECT 'João Jiménez Māori' AS test;

The both show the special chars correctly

But if I do

 UPDATE view SET description = '<p>João Jiménez Māori</p>' where id = 10;

My local shows it like local above but cluster shows it like cluster above.

So the cluster setup for postgres must be different in the way it handles special utf8 characters

[Robert Lyon (robertl-9)]

Will push this problem out to see if elasticsearch, using elasticsearch-php, fixes things

[Robert Lyon (robertl-9)]

Testing this using elasticsearch-php 5.x and elasticsearch 5.x I was able to index things called 'João Jiménez Māori' and search them up again.