Monitor App Curator limits too high
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
StarlingX |
Fix Released
|
High
|
Kevin Smith |
Bug Description
Brief Description
-----------------
The curator limits for indices are too high as elasticsearch will move indices to read only mode after reaching 95% storage high watermark.
Severity
--------
Provide the severity of the defect.
Major: System/Feature is usable but degraded
Steps to Reproduce
------------------
Apply the monitor app and monitor over time as the available storage is used and the cluster status goes from green to yellow as the available storage is filled.
Expected Behavior
------------------
The elasticsearch cluster stays green as indices do not go read only.
Actual Behavior
----------------
The elasticsearch cluster goes yellow and indices enter read only mode.
Reproducibility
---------------
More likely to occur on the system controller in a distributed cloud scenario as indices will fill faster over all index prefixes, but could can occur in a non-dc scenario as well.
System Configuration
-------
All, but more likely in distributed cloud system controller.
Branch/Pull Time/Commit
-------
Latest, as of Jan 6 2020
Last Pass
---------
Never
Timestamp/Logs
--------------
N/A
Test Activity
-------------
Developer Testing
Workaround
----------
Manually set the curator limits lower by using system helm-override command and optionally to also increase the curator cronjob frequency.
Marking as stx.4.0 / high priority - stx-monitor becomes unusable over time.