Reindex tests fail under Elasticsearch 2.x

[Steve McLellan]

After the switch to elasticsearch 2.x for functional tests, several issues became apparent. One of them is that 2 of the tests in functional/test_reindex fail (initially I had seen three failing but I may have been mistaken or one may be sporadic) - test_reindex_with_plugin_and_es and test_reindex_with_plugins. I have disabled these tests as well as test_reindex_with_es in https://review.openstack.org/325487 to unblock the gate.

The tests in question fail with:

FAIL: searchlight.tests.functional.test_reindex.TestReindexing.test_reindex_with_plugin_and_es
tags: worker-1
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/mclellst/openstack/searchlight/searchlight/tests/functional/test_reindex.py", line 206, in test_reindex_with_plugin_and_es
    self.verify_reindex_state(new_index)
  File "/Users/mclellst/openstack/searchlight/searchlight/tests/functional/test_reindex.py", line 67, in verify_reindex_state
    self.assertEqual((2 * self.num_docs), es_docs['hits']['total'])
  File "/Users/mclellst/openstack/searchlight/.tox/py34/lib/python3.4/site-packages/testtools/testcase.py", line 362, in assertEqual
    self.assertThat(observed, matcher, message)
  File "/Users/mclellst/openstack/searchlight/.tox/py34/lib/python3.4/site-packages/testtools/testcase.py", line 447, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: 12 != 8
Ran 3 (+2) tests in 7.512s (+0.427s)
FAILED (id=2000, failures=2 (+2))
error: testr failed (1)

I have not investigated whether this is a sign of an underlying issue or something that changed in the 1 -> 2.x transition.

[Steve McLellan (sjmc7)]

This is a fun one. There are a couple of issues. One is that the index creation tests are happening sufficiently fast that occasionally they end up with the same name (because the name's based on a timestamp with second resolution). The other more consistent issue is to do with document counts; not sure if they're connected problems.

[Steve McLellan (sjmc7)]

Exception formats changed slightly; the duplicate index error message changed slightly and the duplicate alias->index TransportError that occurs on indexing/delete operations changed format.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/326160

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/326160
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=3e6406875b7734cf9c63c1796474a0b42d60b88d
Submitter: Jenkins
Branch: master

commit 3e6406875b7734cf9c63c1796474a0b42d60b88d
Author: Steve McLellan <email address hidden>
Date: Mon Jun 6 17:22:31 2016 -0500

    Fix reindexing conflict handling in es2

    Elasticsearch 2 changed error formats slightly which prevented both the
    duplicate index detection and the 'alias points at two indices'
    detection introduced as part of the zero-downtime reindexing work.

    Patch supports both exception formats and introduces a new test
    for the alias to multiple index code.

    Change-Id: I3e0afe0f6146ab47e3044a8bbb3bca54585e19ba
    Closes-Bug: #1589319

[Steve McLellan (sjmc7)]

I'm fine backporting this since it's low impact. Some of the other ES2 fixes are not so we need to look at them case by case.

[Steve McLellan (sjmc7)]

Worth also noting that we're at the point where nearly all stable/mitaka cherrypicks require manual merges to work around conflicts.

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/searchlight 1.0.0.0b2 development milestone.