Version conflict during re-indexing

[Travis Tripp]

Similar errors for both Swift and Designate.

If you don't specify a type, it doesn't happen. e.g:

 searchlight-manage index sync

If you specify a parent type, then it happens. e.g.

 searchlight-manage index sync --type OS::Designate::Zone

SWIFT:

I first create a bunch of containers objects and folders with this script:
   - https://github.com/openstack/searchlight/blob/master/test-scripts/generate-swift-data.py

Have swift in own resource group:

[resource_plugin:os_swift_account]
reseller_prefix = AUTH_
enabled = True
resource_group_name = searchlight-swift

[resource_plugin:os_swift_container]
enabled = True

[resource_plugin:os_swift_object]
enabled = True

Ran:

searchlight-manage index sync --type OS::Swift::Account

version conflict, current [463591047000000000], provided [463591047000000000]]'}}, {u'index': {u'status': 409, u'_type': u'OS::Swift::Container', u'_id': u'AUTH_4bade81378e6428db0e896db77d68e02/scale_6_USER', u'_index': u'searchlight-swift-2016_05_20_16_25_52', u'error': u'VersionConflictEngineException

http://paste.openstack.org/show/497944/

DESIGNATE:

Created a zone with several records. Successfully indexed using notifications. However, when running

searchlight-manage index sync --type OS::Designate::Zone

I get the following:

2016-06-15 16:26:45.147 ERROR searchlight.cmd.manage [-] Failed to setup index extension os_designate_zone: (u'3 document(s) failed to index.', [{u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'3fc274da-e000-4121-a627-e1de3e7ac3a1', u'_index': u'searchlight-2016_06_15_22_26_44', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][3fc274da-e000-4121-a627-e1de3e7ac3a1]: version conflict, current [466029410000000000], provided [466029410000000000]]'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'6d4bd96d-296d-448b-9050-025fbc3daca3', u'_index': u'searchlight-2016_06_15_22_26_44', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][6d4bd96d-296d-448b-9050-025fbc3daca3]: version conflict, current [466029435000000000], provided [466029435000000000]]'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f', u'_index': u'searchlight-2016_06_15_22_26_44', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f]: version conflict, current [466029435000000000], provided [466029435000000000]]'}}])
2016-06-15 16:26:45.157 INFO elasticsearch [-] DELETE http://127.0.0.1:9200/searchlight-2016_06_15_22_26_44 [status:200 request:0.010s]
2016-06-15 16:26:45.157 DEBUG elasticsearch [-] > None from (pid=55975) log_request_success /usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py:65
2016-06-15 16:26:45.157 DEBUG elasticsearch [-] < {"acknowledged":true} from (pid=55975) log_request_success /usr/local/lib/python2.7/dist-packages/elasticsearch/connection/base.py:66
2016-06-15 16:26:45.158 CRITICAL searchlight [-] BulkIndexError: (u'3 document(s) failed to index.', [{u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'3fc274da-e000-4121-a627-e1de3e7ac3a1', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][3fc274da-e000-4121-a627-e1de3e7ac3a1]: version conflict, current [466029410000000000], provided [466029410000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'6d4bd96d-296d-448b-9050-025fbc3daca3', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][6d4bd96d-296d-448b-9050-025fbc3daca3]: version conflict, current [466029435000000000], provided [466029435000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f]: version conflict, current [466029435000000000], provided [466029435000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}])

2016-06-15 16:26:45.158 TRACE searchlight Traceback (most recent call last):
2016-06-15 16:26:45.158 TRACE searchlight File "/usr/local/bin/searchlight-manage", line 10, in <module>
2016-06-15 16:26:45.158 TRACE searchlight sys.exit(main())
2016-06-15 16:26:45.158 TRACE searchlight File "/opt/stack/searchlight/searchlight/cmd/manage.py", line 380, in main
2016-06-15 16:26:45.158 TRACE searchlight return CONF.command.action_fn(*func_args, **func_kwargs)
2016-06-15 16:26:45.158 TRACE searchlight File "/opt/stack/searchlight/searchlight/cmd/manage.py", line 260, in sync
2016-06-15 16:26:45.158 TRACE searchlight type_list=es_reindex)
2016-06-15 16:26:45.158 TRACE searchlight File "/opt/stack/searchlight/searchlight/elasticsearch/plugins/utils.py", line 144, in reindex
2016-06-15 16:26:45.158 TRACE searchlight target_index=dst_index, query=body)
2016-06-15 16:26:45.158 TRACE searchlight File "/opt/stack/searchlight/searchlight/elasticsearch/plugins/utils.py", line 118, in helper_reindex
2016-06-15 16:26:45.158 TRACE searchlight chunk_size=chunk_size, **kwargs)
2016-06-15 16:26:45.158 TRACE searchlight File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 188, in bulk
2016-06-15 16:26:45.158 TRACE searchlight for ok, item in streaming_bulk(client, actions, **kwargs):
2016-06-15 16:26:45.158 TRACE searchlight File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 160, in streaming_bulk
2016-06-15 16:26:45.158 TRACE searchlight for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
2016-06-15 16:26:45.158 TRACE searchlight File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/__init__.py", line 132, in _process_bulk_chunk
2016-06-15 16:26:45.158 TRACE searchlight raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
2016-06-15 16:26:45.158 TRACE searchlight BulkIndexError: (u'3 document(s) failed to index.', [{u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'3fc274da-e000-4121-a627-e1de3e7ac3a1', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][3fc274da-e000-4121-a627-e1de3e7ac3a1]: version conflict, current [466029410000000000], provided [466029410000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'6d4bd96d-296d-448b-9050-025fbc3daca3', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][6d4bd96d-296d-448b-9050-025fbc3daca3]: version conflict, current [466029435000000000], provided [466029435000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}, {u'index': {u'status': 409, u'_type': u'OS::Designate::RecordSet', u'_id': u'e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f', u'error': u'VersionConflictEngineException[[searchlight-2016_06_15_22_26_44][0] [OS::Designate::RecordSet][e5b63e53-0e36-4ff0-9b70-dbf83a0ffa2f]: version conflict, current [466029435000000000], provided [466029435000000000]]', u'_index': u'searchlight-2016_06_15_22_26_44'}}])
2016-06-15 16:26:45.158 TRACE searchlight

[Travis Tripp (travis-tripp)]

I am also seeing this when swift is in the same resource group. Here is the error:

http://paste.openstack.org/show/516341/

[Steve McLellan (sjmc7)]

Is this specifically happening when you reindex a parent plugin?

[Travis Tripp (travis-tripp)]

Yes, it is when I do:

--type OS::Swift::Account

--type OS::Designate::Zone

[Steve McLellan (sjmc7)]

Haven't 100% confirmed but my spidey senses tell me this is because of the optimization for --type. When you specify --type Designate::Zone searchlight-manage first copies any other doc types over, including in this case OS::Designate::RecordSet. During Zone indexing, the recordsets are once more stored except that since they're the same versions there's a conflict error.

Fix is either to force reindexing all child documents when parents are reindexed (which I'm not particularly fond of) or skip the child indexing under these circumstances. What I'll actually probably do is provide a flag to say "index all children of any --type arguments too".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/332242

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/332242
Committed: https://git.openstack.org/cgit/openstack/searchlight/commit/?id=fa7096232b1a80a486d112b9f98a1306e4798cda
Submitter: Jenkins
Branch: master

commit fa7096232b1a80a486d112b9f98a1306e4798cda
Author: Steve McLellan <email address hidden>
Date: Tue Jun 21 11:38:12 2016 -0500

    Fix --type handling for indexing parent plugins

    When --type is used to index a parent plugin, any child plugins are also
    indexed from their APIs, and then the data's copied into the new index,
    causing conflict errors. This patch simplifies indexing such that child
    plugins are not indexed as a part of indexing the parent since the
    reason for doing that in the past was refactored some time ago. Only
    types specified in --type will be indexed from API; others from an old
    index.

    (See https://review.openstack.org/#/c/295538/)

    This patch removes initial_indexing since it no longer has use, and
    renames setup_data to index_initial_data.

    Patch also changes the index sync diagnostic text to indicate (with a *)
    plugins which will be indexed from existing Elasticsearch data when
    --type is specified.

    Change-Id: Idbec6d3bf663cda7eeb6ab4be19b73d41fa2c8da
    Closes-Bug: #1584148

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/searchlight 1.0.0.0b3 development milestone.