wrong number_of_replicas setting for existing indices when scaling the cluster

[Swann Croiset]

The number_of_replicas is not correct for existing indices after scaling the ES cluster.

This is particularly critical for the index 'kibana-int': when starting with one node and scaling up the cluster to 3 nodes the HA is not ensured (number_of_replicas still to zero) and leads to lost Kibana dashboards if the primary node is down/crashed.

The plugin must re-configure dynamically this setting:
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-update-settings.html

Other indices (log-* and notifications-*) will get the correct number_of_replicas the next day of the scale up but the current and older indices won't, hence there won't be HA.

The scale down case is also impacted in this way: the health of the cluster will be yellow (warning) because the number_of_replicas cannot be honored (greater than the number of available nodes).

[Swann Croiset (swann-w)]

We can use the setting: index.auto_expand_replicas
to automatically adapt the number of replicas based on the number of available nodes

https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/272092

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/272092
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=88e88927f3e128297754d9a38c02e6edeaa78f30
Submitter: Jenkins
Branch: master

commit 88e88927f3e128297754d9a38c02e6edeaa78f30
Author: Swann Croiset <email address hidden>
Date: Mon Jan 25 14:11:39 2016 +0100

    Automatically expand replica numbers for Kibana index

    The patch uses 'auto_expand_replicas' setting to automatically expand the
    number of replicas based on the number of nodes in the cluster.
    This setting ensures HA for Kibana dashboards (if enough nodes are deployed).
    Because dashboards are only few kilobytes they are duplicated on all
    nodes available by specifying the range '0-all'.

    Fixes-bug: #1536689

    DocImpact

    Change-Id: I2643a9e7a18091201ee205149a7316d00acb00ca