Update Elastica to version 2.0.0

[Robert Lyon]

Used for elasticsearch - currently version 0.90.1

[Stéphane Lavoie (lavoie-stephane)]

WE did not install ElasticSearch with our 1.9 Mahara (on MySQL) because it relied on an old version.

And internal search engine of Mahara is really not optimized (lots of double entries in search result pages).

So we're stuck for the moment with no real good search solution.

[Aaron Wells (u-aaronw)]

One complication in this one, is that the Elastica library version is tightly coupled to the Elasticsearch server version. And they're not tested for backwards-compatibility. So updating the Elastica library (which we should do) also requires updating the Elasticsearch server version at the same time.

I'm tempted to say that what we should do, is:

1. Move the search/elasticsearch plugin *out* of Mahara core.
2. Make an updated version of search/elasticsearch for the latest version of Elastica
3. Make the older & newer versions of search/elasticsearch available in the Mahara plugins database, and let people choose which one matches their Elasticsearch server version.

[Dominique-Alain JAN (dajan)]

And wouldn't it be possible to keep the two versions in core and ask the admin to select which version to use with a select menu in the config panel ?

[Aaron Wells (u-aaronw)]

That would be possible, but it require the code to be more complicated. We'd essentially need to write a "wrapper" layer around the Elastica library to account for any changes in its API.

I guess it depends on how much Elastica actually changes its API from one version to the next. We can expect that the 0.9.x versions might be incompatible with the 1.0.x versions, but I haven't actually investigated it in depth.

[Aaron Wells (u-aaronw)]

After further meditation, I think the best thing to do is what I discussed in comment #2. We remove the search/elasticsearch plugin from core and make it an optional plugin hosted in a separate repository. That way we can maintain multiple versions of the plugin for multiple versions of Elastica, and sites can just "drop in" the version that matches their server.

The tricky part here is the process of removing it from core. We need to support two scenarios:

1. Sites that don't use the plugin: Easy peasy, just delete all the plugin's data from the various plugin tables, and drop the search_elasticsearch_queue table.

2. Sites that DO use the plugin: More complicated. We don't want to force them to reindex their site! Ideally the admin will drop in the elasticsearch plugin as part of the upgrade, but we should also support the case where they don't read the README and don't notice the plugin is missing until after they do the upgrade.

2a. Admin drops in elasticsearch plugin before running the upgrade: The code that uninstalls the elasticsearch plugin should check for the presence of the plugin's files, and if it finds them, it assumes the admin has chosen to manually drop in the plugin. In this case, do nothing.

2b. Admin forgets to drop in elasticsearch plugin before running the upgrade: What we want to do is make it possible for them to drop in the plugin at a later date, with no data loss. There are two data loss scenarios here. The first, is there could be unprocessed data in search_elasticsearch_queue. So don't drop that table unless it's empty. The second data loss scenario, is that currently the only way to get the elasticsearch triggers to be recreated, is to press the "reindex" button. So, we should add a separate button to just recreate the elasticsearch triggers without also forcing a reindex.

[Aaron Wells (u-aaronw)]

It's worth noting, Son tested out a local copy of the site with the latest version of Elastica (1.3) and the 0.9 version of Elasticsearch, and it appeared to work fine.

So in light of that, and of the difficulties of dropping the plugin from core, we might just upgrade the bundled Elastica version.

[Aaron Wells (u-aaronw)]

Okay, I've tested Elastica 2.0.0 (the latest) with Elasticsearch 1.5 (the latest) and Elasticsearch 0.9.

It did not work out of the box with Elasticsearch 0.9, because Elastica 2.0 uses the "post_filter" query parameter when you do \Elastica\Query->set_filter(), and Elasticsearch 0.9 apparently didn't support that yet. I was able to get it to work in both, however, by tweaking our code so that it uses a \Elastica\Query\Filtered instead.

I'll push that up to Gerrit.

[Mahara Bot (dev-mahara)]

Patch for "master" branch: https://reviews.mahara.org/4787

[Mahara Bot (dev-mahara)]

Patch for "master" branch: https://reviews.mahara.org/4786

[Mahara Bot (dev-mahara)]

Reviewed: https://reviews.mahara.org/4786
Committed: https://git.nzoss.org.nz/mahara/mahara/commit/2455c04994dfff665d5940452ae4c87f2512d865
Submitter: Aaron Wells (<email address hidden>)
Branch: master

commit 2455c04994dfff665d5940452ae4c87f2512d865
Author: Aaron Wells <email address hidden>
Date: Fri May 22 14:07:35 2015 +1200

Upgrading Elastica to version 2.0.0

Bug 1384498

Change-Id: Ib3241f69ff92c6ec11cdf69425bd44ea3d728f2a

[Mahara Bot (dev-mahara)]

Reviewed: https://reviews.mahara.org/4787
Committed: https://git.nzoss.org.nz/mahara/mahara/commit/4391209470e30dc91eb1640704944aac03224675
Submitter: Robert Lyon (<email address hidden>)
Branch: master

commit 4391209470e30dc91eb1640704944aac03224675
Author: Aaron Wells <email address hidden>
Date: Fri May 22 13:57:04 2015 +1200

Patching search/elasticsearch to work with elasticsearch 0.9 and 1.5

Bug 1384498: Query->set_filter() uses a "post_filter" param, which
isn't supported in Elasticsearch 0.9. But if we use a \Query\Filtered,
it works in both versions.

Change-Id: Ie82edb35740a390b89c19a430fea36a4abf6b69f