Cannot use Aggregator Transformer with retention_time without also having size-based flush

Bug #1531626 reported by Kevin McDonald on 2016-01-06
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Ceilometer
Fix Released
Medium
Kevin McDonald
Kilo
Fix Released
Undecided
gordon chung
Liberty
Fix Committed
Undecided
gordon chung

Bug Description

It appears that setting retention_time without setting size will cause samples to get flushed after 2 samples instead of when retention_time is reached because the default for size is 1. That seems like unintuitive behavior.

This is the example given for configuring an aggregator transformer using the retention time option:
---
transformers:
    - name: "aggregator"
      parameters:
          retention_time: 60
          resource_metadata: last
---

This example does not work. It will always emit the aggregated samples that it has collected after it receives a second sample. This is because size defaults to 1.

Mitigations:
 * If you only want aggregated samples to be flushed by a time-based metric then you'll have to set size to a sufficiently large value

Proposed fix:

I propose that this configuration should flush samples every 60 seconds, regardless of how many samples are collected:
---
transformers:
    - name: "aggregator"
      parameters:
          retention_time: 60
          resource_metadata: last
---
Results in size = None, retention_time = 60 seconds

If neither size or retention_time is specified, the behavior should stay the same: size defaulting to one (no aggregation). I feel like this is incorrect behavior, but for the sake of backwards compatibility that behavior should stay the same:
---
transformers:
    - name: "aggregator"
      parameters:
          resource_metadata: last
---
Results in size = 1, retention_time = None

Obviously, specifying both retention_time and size should use both as criteria to trigger a flush:
---
transformers:
    - name: "aggregator"
      parameters:
          size: 10
          retention_time: 60
          resource_metadata: last
---
Results in size = 10, retention_time = 60 seconds

Ref:

http://docs.openstack.org/admin-guide-cloud/telemetry-data-collection.html#telemetry-pipeline-configuration

https://github.com/openstack/ceilometer/blob/f87edaa1e01b62e5316039d9977a2d1a11b6271e/ceilometer/transformer/conversions.py#L311

description: updated
description: updated
Changed in ceilometer:
assignee: nobody → Rohit Jaiswal (rohit-jaiswal-3)

Fix proposed to branch: master
Review: https://review.openstack.org/264376

Changed in ceilometer:
assignee: Rohit Jaiswal (rohit-jaiswal-3) → Kevin McDonald (k3vinmcdonald)
status: New → In Progress
gordon chung (chungg) on 2016-01-10
Changed in ceilometer:
importance: Undecided → Medium

Reviewed: https://review.openstack.org/264376
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=5a9a41563e9c26fb91f4d5ebf863bf24bf395237
Submitter: Jenkins
Branch: master

commit 5a9a41563e9c26fb91f4d5ebf863bf24bf395237
Author: Kevin McDonald <email address hidden>
Date: Wed Jan 6 15:52:55 2016 -0600

    Changes aggregator transformer to allow retention_time w/o size

    When using an aggregator transformer in pipeline.yaml that
    looks like this:
    ---
    transformers:
        - name: "aggregator"
          parameters:
              retention_time: 60
              resource_metadata: last
    ---
    'size' will automatically be defaulted to 1. That means that
    aggregation will not happen as expected with a retention_time of
    60 seconds.

    This fix will only default sample size to 1 if neither
    retention_time or size are defined.

    Change-Id: I4a3aa0f6de26173e6f9383d570ff2cf13d367e38
    Closes-Bug: 1531626

Changed in ceilometer:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/266075
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=fd54d10f6d35ea00feee67d222a90e4ad3951faa
Submitter: Jenkins
Branch: stable/liberty

commit fd54d10f6d35ea00feee67d222a90e4ad3951faa
Author: Kevin McDonald <email address hidden>
Date: Wed Jan 6 15:52:55 2016 -0600

    Changes aggregator transformer to allow retention_time w/o size

    When using an aggregator transformer in pipeline.yaml that
    looks like this:
    ---
    transformers:
        - name: "aggregator"
          parameters:
              retention_time: 60
              resource_metadata: last
    ---
    'size' will automatically be defaulted to 1. That means that
    aggregation will not happen as expected with a retention_time of
    60 seconds.

    This fix will only default sample size to 1 if neither
    retention_time or size are defined.

    Change-Id: I4a3aa0f6de26173e6f9383d570ff2cf13d367e38
    Closes-Bug: 1531626
    (cherry picked from commit 5a9a41563e9c26fb91f4d5ebf863bf24bf395237)

This issue was fixed in the openstack/ceilometer 6.0.0.0b2 development milestone.

This issue was fixed in the openstack/ceilometer 5.0.2 release.

Reviewed: https://review.openstack.org/266076
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=b968f5fba05e439f9c65d504741bb8ab236e756f
Submitter: Jenkins
Branch: stable/kilo

commit b968f5fba05e439f9c65d504741bb8ab236e756f
Author: Kevin McDonald <email address hidden>
Date: Wed Jan 6 15:52:55 2016 -0600

    Changes aggregator transformer to allow retention_time w/o size

    When using an aggregator transformer in pipeline.yaml that
    looks like this:
    ---
    transformers:
        - name: "aggregator"
          parameters:
              retention_time: 60
              resource_metadata: last
    ---
    'size' will automatically be defaulted to 1. That means that
    aggregation will not happen as expected with a retention_time of
    60 seconds.

    This fix will only default sample size to 1 if neither
    retention_time or size are defined.

    Change-Id: I4a3aa0f6de26173e6f9383d570ff2cf13d367e38
    Closes-Bug: 1531626
    (cherry picked from commit 5a9a41563e9c26fb91f4d5ebf863bf24bf395237)

This issue was fixed in the openstack/ceilometer 2015.1.4 release.

This issue was fixed in the openstack/ceilometer 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Other bug subscribers