StackLight

The log_collector service is blocked when the Elasticsearch cluster is too slow

Bug #1617211 reported by Simon Pasquier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StackLight
Fix Released
Medium
Simon Pasquier
StackLight 1.0.0
0.10
Fix Released
Undecided
Unassigned
StackLight 0.10.2

Bug Description

In some environments (especially using slow HDD drives), the Elasticsearch backends may fail to ingest logs fast enough. As a result the log_collector service running on the controller nodes are blocked.

OpenStack Infra (hudson-openstack)
Changed in lma-toolchain:
assignee: LMA-Toolchain Fuel Plugins (mos-lma-toolchain) → Simon Pasquier (simon-pasquier)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (master)

Reviewed: https://review.openstack.org/356344
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=83db24f5496d777232083f04f997fd60b9ab8663
Submitter: Jenkins
Branch: master

commit 83db24f5496d777232083f04f997fd60b9ab8663
Author: Swann Croiset <email address hidden>
Date: Wed Aug 17 11:38:10 2016 +0200

    Increase the Elasticsearch bulk size when required

    In some environments (especially using slow HDD drives), the
    Elasticsearch backends may fail to ingest logs fast enough. As a result
    the log_collector service running on the controller nodes are blocked.

    To alleviate this issue, this change increases the bulk size for nodes
    that generate lots of logs:
    - controllers which run OpenStack API services in addition to Pacemaker.
    - all nodes when the environment's log level is set to debug.

    In such cases, the flush_count parameter is increased to 100 (instead of
    10 by default).

    Change-Id: Ifdfbcb8ff0292f695dee4deab45560f126bde242
    Closes-Bug: #1617211

Changed in lma-toolchain:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-plugin-lma-collector (stable/0.10)

Fix proposed to branch: stable/0.10
Review: https://review.openstack.org/379978

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-plugin-lma-collector (stable/0.10)

Reviewed: https://review.openstack.org/379978
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=55b1c6daa4cf44d071fd22875be50c74779dd67d
Submitter: Jenkins
Branch: stable/0.10

commit 55b1c6daa4cf44d071fd22875be50c74779dd67d
Author: Swann Croiset <email address hidden>
Date: Wed Aug 17 11:38:10 2016 +0200

    Increase the Elasticsearch bulk size when required

    In some environments (especially using slow HDD drives), the
    Elasticsearch backends may fail to ingest logs fast enough. As a result
    the log_collector service running on the controller nodes are blocked.

    To alleviate this issue, this change increases the bulk size for nodes
    that generate lots of logs:
    - controllers which run OpenStack API services in addition to Pacemaker.
    - all nodes when the environment's log level is set to debug.

    In such cases, the flush_count parameter is increased to 100 (instead of
    10 by default).

    Change-Id: Ifdfbcb8ff0292f695dee4deab45560f126bde242
    Closes-Bug: #1617211
    (cherry picked from commit 83db24f5496d777232083f04f997fd60b9ab8663)

Simon Pasquier (simon-pasquier)
Changed in lma-toolchain:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.