Arithmetic transformer not working on multiple meters

[Claudio]

The poll_and_publish method of a PollingTask object (file agent.py) on each iteration gets the samples from the current pollster and injects them into the pipeline. Here, samples are transformed before getting published.

The arithmetic transformer needs to be applied on samples from every meters on which the transformer is defined. But in the poll_and_publish method, only samples from a single meter per iteration are polled (and fed to the arithmetic transformer), so I get this error:
"Unable to perform calculation, not all of {_disk_write_bytes_ESC, _disk_read_bytes_ESC} are present"

Here is an extract from my pipeline.yaml file:

sources:
   [...]
    - name: disk_source
      interval: 600
      meters:
          - "disk.read.bytes"
          - "disk.read.requests"
          - "disk.write.bytes"
          - "disk.write.requests"
      sinks:
          - disk_sink
          - disk_read_write_sink
    [...]
sinks:
   [...]
    - name: disk_read_write_sink
      transformers:
          - name: "arithmetic"
            parameters:
                target:
                    name: "disk.read.write.bytes"
                    unit: "B"
                    type: "cumulative"
                    expr: "2*$(disk.read.bytes) + 2*$(disk.write.bytes)"
     [...]

[Nejc Saje (nejc-saje)]

I think it'd be a good thing to change the tests in pipeline_base that test transformers to use polling tasks, not pipelines directly, so this issue would surface there.

As for the fix, I'm thinking it'd probably be about grouping pollster_matches by source and then encompassing all of the pollsters for the same source in the same publish context.

Looking forward to reviewing the patch Igor :-)

[Claudio (claudio-pupparo)]

I have created a fix for this bug, it's in the attachment.
There are also two tests that reference the older pollster_matches variable, so I think they must be modified too:
./ceilometer/tests/agentbase.py:364: pollsters = polling_tasks.get(60).pollster_matches
./ceilometer/tests/agentbase.py:748: pollster = list(polling_task.pollster_matches)[0][1]

[Claudio (claudio-pupparo)]

The solution is similar to the one proposed by Nejc ;)

[Igor Degtiarov (idegtiarov)]

Thanks Nejc and Claudio. Will look at your advices Claudio.

[Igor Degtiarov (idegtiarov)]

I cannot reproduce that bug on my local devstack.

All meters are collected and arithmetic transformer is working in proper way. Claudio could you please explain in more details what case is caused to the bug that you registered.

[Nejc Saje (nejc-saje)]

I was able to reproduce it locally by using the exact same config as Claudio in his mailing list e-mail: http://lists.openstack.org/pipermail/openstack/2014-November/010384.html . What config did you use Igor? It would be interesting to see in which case it still works.

The bug is a regression stemming from the refactoring done on the way the pollsters are run. They used to be run in the same publishing context, now they're not, and they should be again.

[Igor Degtiarov (idegtiarov)]

Nejc thank you for quick answer! The most interesting thing is that my config is the same as in Claudio mail, and it works as it should be.

I try to guess how I can reproduce it? Nejc could you please explain how you caught it?

[Nejc Saje (nejc-saje)]

Are you using the latest Ceilometer code? Because judging by the code, it shouldn't work.

In https://github.com/openstack/ceilometer/blob/master/ceilometer/agent.py#L122 , each pollster is called in its own publish context. When that context exits, it calls flush() on transformers, https://github.com/openstack/ceilometer/blob/master/ceilometer/pipeline.py#L74 , and the arithmetic transformer does its checks and calculations in the flush() method: https://github.com/openstack/ceilometer/blob/master/ceilometer/transformer/arithmetic.py#L102

[Claudio (claudio-pupparo)]

I forgot to mention, that I've done tests under branches stable/juno and master, and in each case I've got the bug.
The reason is the one explained by Nejc.

[Claudio (claudio-pupparo)]

Unfortunately I've found that my solution doesn't work on arithmetic transformers applied to meters from different sources.
This is due to how a pipeline is defined.
A pipeline is a coupling between a (single) source and a sink. When a pipeline is flushed, only meters from the source "belonging" to the pipeline will be published.
So an arithmetic transformer defined on meters from different sources gives "Unable to perform calculation, not all of [...] are present"

[Nejc Saje (nejc-saje)]

Yes, Claudio, that is the limitation of the current implementation. Since pipelines weren't always so separate, the limitation used to only be to meters with the same interval. The note in http://docs.openstack.org/developer/ceilometer/configuration.html#multi-meter-arithmetic-transformer should be changed to say "The calculation is limited to meters from the same source.".

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/139037

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/139037
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=21935cb0c1074233a0782795d3cdf7d2798fd67d
Submitter: Jenkins
Branch: master

commit 21935cb0c1074233a0782795d3cdf7d2798fd67d
Author: Igor Degtiarov <email address hidden>
Date: Tue Dec 9 15:51:32 2014 +0200

    Encompassing one source pollsters with common context

    After per-source separation of static resources & discovery samples are
    publishing with context of separate pollster, so if it is used arithmetic
    transformer which handle different meters, it will fail.

    Now samples are publishing from common source context, after all
    source meters have been collected.

    Change-Id: I0d186247b4b87fd94317318c180bdfb9c6da70fc
    Closes-bug: #1394228

[Claudio (claudio-pupparo)]

Good job Igor! ;)

[Igor Degtiarov (idegtiarov)]

Thank you Claudio! :D
-- Igor

On Fri, Dec 12, 2014 at 1:18 PM, Claudio <email address hidden> wrote:
> Good job Igor! ;)
>
> --
> You received this bug notification because you are a bug assignee.
> Matching subscriptions: Igor Degtiarov, Igor Degtiarov
> https://bugs.launchpad.net/bugs/1394228
>
> Title:
> Arithmetic transformer not working on multiple meters
>
> Status in OpenStack Telemetry (Ceilometer):
> Fix Committed
>
> Bug description:
> The poll_and_publish method of a PollingTask object (file agent.py) on
> each iteration gets the samples from the current pollster and injects
> them into the pipeline. Here, samples are transformed before getting
> published.
>
> The arithmetic transformer needs to be applied on samples from every meters on which the transformer is defined. But in the poll_and_publish method, only samples from a single meter per iteration are polled (and fed to the arithmetic transformer), so I get this error:
> "Unable to perform calculation, not all of {_disk_write_bytes_ESC, _disk_read_bytes_ESC} are present"
>
> Here is an extract from my pipeline.yaml file:
>
> sources:
> [...]
> - name: disk_source
> interval: 600
> meters:
> - "disk.read.bytes"
> - "disk.read.requests"
> - "disk.write.bytes"
> - "disk.write.requests"
> sinks:
> - disk_sink
> - disk_read_write_sink
> [...]
> sinks:
> [...]
> - name: disk_read_write_sink
> transformers:
> - name: "arithmetic"
> parameters:
> target:
> name: "disk.read.write.bytes"
> unit: "B"
> type: "cumulative"
> expr: "2*$(disk.read.bytes) + 2*$(disk.write.bytes)"
> [...]
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ceilometer/+bug/1394228/+subscriptions