Transformer of type "unit_conversion": "scale" parameter

[Viktor Horvath]

For testing purposes, I tried to create a very simple new transformer, converting cpu in nanoseconds to "cpu_sec" in seconds. The "scale" parameter behaves differently : if it is a numeric value, it is a scale factor for the volume. But if it's a string, it must be an expression containing "volume". In other words,

scale: 0.000000001

is not the same as

scale: "0.000000001"

but it is the same as

scale: "volume * 0.000000001"

In transformer/conversions.py I find these lines :
        return ((eval(scale, {}, ns) if isinstance(scale, six.string_types)
                 else s.volume * scale) if scale else s.volume)
If "scale" is a string, its value is taken as such, not multiplied with "s.volume".

I find this not very intuitive, particularly because the "cpu_sink" - the one that calculates "cpu_util" - has a scale string which simply becomes a factor. But the cpu_util is a "rate_of_change" transformer :
scale: "100.0 / (10**9 * (resource_metadata.cpu_number or 1))"

Maybe you want to consider either putting this detail into the documentation or adding the factor "s.volume" into conversions.py for a "scale" string ?

[Viktor Horvath (viktorhorvath)]

I'll prepare a tiny patch for conversions.py, because from the documentation example on
http://docs.openstack.org/developer/ceilometer/configuration.html#unit-conversion-transformer
it's clear that this is supposed to work:

Sample configuration:

transformers:
- name: "unit_conversion"
  parameters:
      target:
          name: "disk.kilobytes"
          unit: "KB"
          scale: "1.0 / 1024.0"

[Viktor Horvath (viktorhorvath)]

OK, it's not that easy due to the inheritance between unit_conversion and rate_of_change. The "scale" parameter sometimes means a factor, and sometimes it means a formula / expression.

For a unit_conversion transformer:
scale: 0.2
and
scale: "volume * 0.02"
are equivalent, but in the second case, a better name would be "formula" or "expr".
scale: "0.02" just produces data points of 0.02. Also, with just a scale factor, you couldn't run the Celsius-Fahrenheit example, because "(volume * 1.8) + 32" is a real formula and cannot be reduced to a scale factor.

BUT, for a rate_of_change, "volume" does not make sense any more, the delta between two data points becomes an implicit factor. So,
scale: 0.02
does not work at all, because values do get multiplied by "volume" in the superclass;
scale:"0.02"
works, the value gets only multiplied by the delta.

For proper clean-up, one would need to introduce a new parameter "formula" or "expr" and separate the two use cases. Maybe one would even break the inheritance, because rate_of_change is not really a special case of a unit_conversion.

Or we just fix the documentation quoted in my comment above. What do you think?

[Nejc Saje (nejc-saje)]

Thanks for researching this problem Viktor! Sorry for late response, we've been busy as the Juno release is nearing.

The main point to keep in mind while we search for a solution is that existing configurations must remain working. I would vote for changing the scaling so that both strings and integers are considered a factor by which the volume is multiplied.

We would however need to transparently (and undocumented-ly) support the old behaviour, where the volume can be passed in as explicit. That would need to be detected and in that case, the scaling factor shouldn't be multiplied with volume.

In addition, the rate_of_change transformer should be modified to ensure that with either string or integer scale, it's the delta that's multiplied, not the volume.

The expr/formula approach and the Celsius-Fahrenheit example would be better suited for the new Arithmetic transformer (http://docs.openstack.org/developer/ceilometer/configuration.html#multi-meter-arithmetic-transformer).

Cheers,
Nejc