Elasticsearch failure when disk full

[Ivan Lozgachev]

This bug is created to track https://github.com/elastic/elasticsearch/issues/12055

Environment:
3 controllers
15 compute + ceph nodes
1 elasticsearch node
1 influxdb node

How to reproduce:
1. Login to elasticsearch node
2. "df -h" to check how much free space exists
3. cd /opt/es-data/
4. Use "fallocate" to completely fill disk space
5. Wait for some time (5 minutes is enough)
6. Cleanup disk

Expected result:
elasticsearch and Kibana works without errors after cleanup

Actual result:
Kibana stopped to show "Log messages over time" statistic

[Ivan Lozgachev (ilozgachev)]

This found on Fuel 8.0 build 552, LMA toolchain from origin/master

[Simon Pasquier (simon-pasquier)]

This won't be available before LMA 1.0 since as far as I understand, this would require the Elasticsearch-Kibana plugin to install Elasticsearch 2.x instead of 1.7.x.

[Simon Pasquier (simon-pasquier)]

The workaround for LMA 0.9 and before is to delete the .recovery file(s) inside the translog folder(s) on the Elasticsearch node(s).

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/322222
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=0c696459c9ae7c7051cd91c1a577d5df9ddd7fc4
Submitter: Jenkins
Branch: master

commit 0c696459c9ae7c7051cd91c1a577d5df9ddd7fc4
Author: Swann Croiset <email address hidden>
Date: Fri May 27 13:57:45 2016 +0200

    Upgrade to Kibana 4.5.1 / Elasticsearch 2.3.3

    DocImpact Kibana Screenshot and ES upgrade

    Fixes-bug: #1545739

    Change-Id: I068ec6e6981b9b7729d0041be65760206790f615