Improve the InfluxDB performances

Bug #1581369 reported by Simon Pasquier on 2016-05-13
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
StackLight
Wishlist
Swann Croiset

Bug Description

The performances of InfluxDB could be improved in several ways:
- Batch more points when writing. We currently batch up to 100 datapoints per write request while it is generally recommended to batch 5,000 points.
  Expected improvement: higher write throughput.
- Use a second resolution (instead of milliseconds) for datapoints. Realistically we don't need the timestamps to have a ms precision.
  Expected improvement: less memory & storage consumption and improved query response times.

description: updated

Fix proposed to branch: master
Review: https://review.openstack.org/318666

Changed in lma-toolchain:
assignee: nobody → Simon Pasquier (simon-pasquier)
status: New → In Progress

Reviewed: https://review.openstack.org/318666
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=567562faaf6ac60880eec4ab9e66154b70f2ad9e
Submitter: Jenkins
Branch: master

commit 567562faaf6ac60880eec4ab9e66154b70f2ad9e
Author: Simon Pasquier <email address hidden>
Date: Thu May 19 15:25:43 2016 +0200

    Increase the number of points per InfluxDB batch

    This change increases the maximum number of points that are sent in a
    single request. InfluxDB recommends to have a batch size of 5,000 so
    this is now the default configuration value. Note that the InfluxDB
    accumulator will flush the data either when it holds 5,000 points or
    when it hasn't data for at least 5 seconds.

    Change-Id: If07b7d285d216855997254952ca6d7511cff65ec
    Partial-Bug: #1581369

Fix proposed to branch: master
Review: https://review.openstack.org/380147

Changed in lma-toolchain:
assignee: Simon Pasquier (simon-pasquier) → Swann Croiset (swann-w)

Reviewed: https://review.openstack.org/380147
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-lma-collector/commit/?id=2cc44ddba0065bb01a96e4b1d69d97d4a5c5a060
Submitter: Jenkins
Branch: master

commit 2cc44ddba0065bb01a96e4b1d69d97d4a5c5a060
Author: Simon Pasquier <email address hidden>
Date: Thu May 19 15:25:43 2016 +0200

    Increase the number of points per InfluxDB batch

    This change improves the InfluxDB write performances by increasing to
    500 the maximum number of points that are sent per InfluxDB request.
    InfluxDB recommends to have a batch size of 5,000 but it cannot be the default
    configuration value due to the fixed sized of Heka messages (256K currently)
    which leads to silently discard metrics.
    Note that the InfluxDB accumulator will flush the data either when
    it holds 500 points or when it hasn't data for at least 5 seconds.

    Co-Authored-By: Swann Croiset <email address hidden>

    Change-Id: I7d238375dc0c231782983fc4901c9a32936fb08a
    Partial-Bug: #1581369

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers