Elasticsearch failure when disk full

Bug #1545739 reported by Ivan Lozgachev on 2016-02-15
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Swann Croiset

Bug Description

This bug is created to track https://github.com/elastic/elasticsearch/issues/12055

3 controllers
15 compute + ceph nodes
1 elasticsearch node
1 influxdb node

How to reproduce:
1. Login to elasticsearch node
2. "df -h" to check how much free space exists
3. cd /opt/es-data/
4. Use "fallocate" to completely fill disk space
5. Wait for some time (5 minutes is enough)
6. Cleanup disk

Expected result:
elasticsearch and Kibana works without errors after cleanup

Actual result:
Kibana stopped to show "Log messages over time" statistic

Ivan Lozgachev (ilozgachev) wrote :

This found on Fuel 8.0 build 552, LMA toolchain from origin/master

Changed in lma-toolchain:
status: New → Confirmed
assignee: nobody → LMA-Toolchain Fuel Plugins (mos-lma-toolchain)
Simon Pasquier (simon-pasquier) wrote :

This won't be available before LMA 1.0 since as far as I understand, this would require the Elasticsearch-Kibana plugin to install Elasticsearch 2.x instead of 1.7.x.

Changed in lma-toolchain:
milestone: none → 1.0.0
Simon Pasquier (simon-pasquier) wrote :

The workaround for LMA 0.9 and before is to delete the .recovery file(s) inside the translog folder(s) on the Elasticsearch node(s).

Changed in lma-toolchain:
importance: Undecided → Medium
Swann Croiset (swann-w) on 2016-02-25
summary: - Kibana failure when disk full
+ Elasticsearch failure when disk full
Changed in lma-toolchain:
milestone: 1.0.0 → none
Swann Croiset (swann-w) on 2016-05-30
Changed in lma-toolchain:
milestone: none → 0.10.0
Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Swann Croiset (swann-w)
status: Confirmed → In Progress

Reviewed: https://review.openstack.org/322222
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-elasticsearch-kibana/commit/?id=0c696459c9ae7c7051cd91c1a577d5df9ddd7fc4
Submitter: Jenkins
Branch: master

commit 0c696459c9ae7c7051cd91c1a577d5df9ddd7fc4
Author: Swann Croiset <email address hidden>
Date: Fri May 27 13:57:45 2016 +0200

    Upgrade to Kibana 4.5.1 / Elasticsearch 2.3.3

    DocImpact Kibana Screenshot and ES upgrade

    Fixes-bug: #1545739

    Change-Id: I068ec6e6981b9b7729d0041be65760206790f615

Changed in lma-toolchain:
status: In Progress → Fix Committed
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers