equivalent of drawNullAsZero flag required for raw data

[cbrinley]

the flag drawNullAsZero works well for graphical representation of data. When a user wishes to receive textual version of data this flag is of no help. The bug proposes a new function called transformNull() that allow Null/None data to be transformed to a static value. This can be combined with other filter functions to appropriately retrieve only "valid" data from graphite.

diff attached.

function help text:

 Takes a metric or wild card seriesList and an optional value
  to transform Nulls to. Default is 0. This method compliments
  drawNullAsZero flag in graphical mode but also works in text only
  mode.
  Example:

  .. code-block:: none

    &target=transformNull(webapp.pages.*.views,-1)

  This would take any page that didn't have values and supply negative 1 as a default.
  Any other numeric value may be used as well.

[Sandy Walsh (sandy-walsh)]

Thanks for this. I would like the readdress the motivation ... I was getting a ton of None values when a &format=json query was selected. I was hoping to not have these None values returned at all from the query (so I wouldn't have to filter them in the client.)

Would I use this patch with the currentAbove(xxx, 0) function (I'm assuming I can nest functions?)

[cbrinley (cbrinley)]

yes the function by its self will just change the value of null to say 0,
when combined with currentAbove you can effectively filter those out by
setting the transform value to something you wouldn't normally get from
your metric like a negative value or 0 if that is effective in your use
case.

use would look like this: currentAbove(tranformNull(metric,0),0)

On Wed, Mar 21, 2012 at 1:26 PM, Sandy Walsh <email address hidden>wrote:

> Thanks for this. I would like the readdress the motivation ... I was
> getting a ton of None values when a &format=json query was selected. I
> was hoping to not have these None values returned at all from the query
> (so I wouldn't have to filter them in the client.)
>
> Would I use this patch with the currentAbove(xxx, 0) function (I'm
> assuming I can nest functions?)
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/961374
>
> Title:
> equivalent of drawNullAsZero flag required for raw data
>
> Status in Graphite - Enterprise scalable realtime graphing:
> New
>
> Bug description:
> the flag drawNullAsZero works well for graphical representation of
> data. When a user wishes to receive textual version of data this flag
> is of no help. The bug proposes a new function called transformNull()
> that allow Null/None data to be transformed to a static value. This
> can be combined with other filter functions to appropriately retrieve
> only "valid" data from graphite.
>
>
> diff attached.
>
> function help text:
>
> Takes a metric or wild card seriesList and an optional value
> to transform Nulls to. Default is 0. This method compliments
> drawNullAsZero flag in graphical mode but also works in text only
> mode.
> Example:
>
> .. code-block:: none
>
> &target=transformNull(webapp.pages.*.views,-1)
>
> This would take any page that didn't have values and supply negative 1
> as a default.
> Any other numeric value may be used as well.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/graphite/+bug/961374/+subscriptions
>

--

Chris Brinley
219.262.0066

[Michael Leinartas (mleinartas)]

Thanks for the new function, this has been merged in r727

[Michael Leinartas (mleinartas)]

Sandy - removeAboveValue is what Chris intended to write I think, but unfortunately it wont work in this case as the way it removes values is setting them to None which will still leave your JSON output polluted with Nones.

There's a lot of logic in Graphite that depends on the the data having even steps between data points so implementing this with a function wont work. It could instead be added as another format (sparseJson?) or as an option applicable to the json and csv formats.

If you'd like to see this, file a bug so that we can keep track of the feature request

[cbrinley (cbrinley)]

So admittedly it seems I may have misunderstood the intent. Based on the
use of currentAbove function i assumed, first mistake :), only the last
value aka "current" was of interest. To Michael's point if you'r looking to
simply prune your list of these null values AND time stamps this
approach provided will not work.

Given some series [(t1,100) (t2,Null) (t3,20)]
You cannot end up with a series that looks like this: [(t1,100) (t3,20)]
via any function set today.

However if your interest was the last data point and only receiving it if
it is not null the provided approach can work. You will receive an empty
json list in cases where last value was null.

The new method transformNull doesn't get you much if you need to exclude
values that are null. If however your goals is to have a numeric value
(0,-1,etc) for something that was null this function may help.

Either way if you need to out right exclude those values then some consumer
side processing will be required. Sorry if this just left you back in the
same spot.

Regards,

On Fri, Mar 23, 2012 at 1:27 AM, Michael Leinartas <
<email address hidden>> wrote:

> Sandy - removeAboveValue is what Chris intended to write I think, but
> unfortunately it wont work in this case as the way it removes values is
> setting them to None which will still leave your JSON output polluted
> with Nones.
>
> There's a lot of logic in Graphite that depends on the the data having
> even steps between data points so implementing this with a function wont
> work. It could instead be added as another format (sparseJson?) or as
> an option applicable to the json and csv formats.
>
> If you'd like to see this, file a bug so that we can keep track of the
> feature request
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/961374
>
> Title:
> equivalent of drawNullAsZero flag required for raw data
>
> Status in Graphite - Enterprise scalable realtime graphing:
> Fix Committed
>
> Bug description:
> the flag drawNullAsZero works well for graphical representation of
> data. When a user wishes to receive textual version of data this flag
> is of no help. The bug proposes a new function called transformNull()
> that allow Null/None data to be transformed to a static value. This
> can be combined with other filter functions to appropriately retrieve
> only "valid" data from graphite.
>
>
> diff attached.
>
> function help text:
>
> Takes a metric or wild card seriesList and an optional value
> to transform Nulls to. Default is 0. This method compliments
> drawNullAsZero flag in graphical mode but also works in text only
> mode.
> Example:
>
> .. code-block:: none
>
> &target=transformNull(webapp.pages.*.views,-1)
>
> This would take any page that didn't have values and supply negative 1
> as a default.
> Any other numeric value may be used as well.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/graphite/+bug/961374/+subscriptions
>

--

Chris Brinley
219.262.0066