ElasticSearch failed to scale down from 3 to 1 node

[Ivan Lozgachev]

Steps to reproduce:

1. Deploy cluster with 3 controllers, 11 compute + ceph, 3 influxdb and 3 elasticsearch nodes
2. Remove 2 elasticsearch nodes.
3. Deploy changes.
4. Check ElasticSearch cluster status in Grafana

Expected result: cluster status is ok/green

Actual result: cluster status is warn/yellow; number of nodes 3 (should be 1)

root@node-1:/var/log/elasticsearch/es-01# curl node-1:9200/_cat/indices?v
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open notification-2016.03.12 5 2
green open kibana-int 5 0 2 0 26.1kb 26.1kb
red open log-2016.03.13 5 2
red open notification-2016.03.13 5 2
red open log-2016.03.12 5 2

I tried to "sudo service elasticsearch-es-01 restart" to recover cluster, but this didn't help and number of nodes in Grafana become N/A state.

[Ivan Lozgachev (ilozgachev)]

Found on Fuel 8.0 build 570, LMA toolchain compiled from origin/master

[Swann Croiset (swann-w)]

This is known. A manual operation is required to reduce the number of replicas for existing indices:

curl -XPUT <ELASTICSEARCH_VIP>:9200/log-<DATE>/_settings -d '
{ "index": { "number_of_replicas": 2 } }'

{"acknowledged":true}

[Swann Croiset (swann-w)]

I meant number_of_replicas set to 0 instead of 2 when you have only one node

[Patrick Petit (patrick-michel-petit)]

This should at least be captured in the doc before we release 0.9.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/292491

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/292491
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=bc9efd93decb2516310398d5c05f1a59f761bec9
Submitter: Jenkins
Branch: master

commit bc9efd93decb2516310398d5c05f1a59f761bec9
Author: Swann Croiset <email address hidden>
Date: Mon Mar 14 17:25:16 2016 +0100

    Add documentation for cluster operations

    Change-Id: Ic090ff7117b7393a9192718c7abab4e089af8dd8
    Related-Bug: #1556576

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/0.9
Review: https://review.openstack.org/294663

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/294663
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=85e779cac26d2e2f243ae13b49dba9c24befd49b
Submitter: Jenkins
Branch: stable/0.9

commit 85e779cac26d2e2f243ae13b49dba9c24befd49b
Author: Swann Croiset <email address hidden>
Date: Mon Mar 14 17:25:16 2016 +0100

    Add documentation for cluster operations

    Change-Id: Ic090ff7117b7393a9192718c7abab4e089af8dd8
    Related-Bug: #1556576
    (cherry picked from commit bc9efd93decb2516310398d5c05f1a59f761bec9)