Grafana status boxes are flapping sometimes

[Swann Croiset]

I've observed that a service which is OKAY can flap to CRITICAL sometimes. Depending on when the request is made, the "group by" returns 'null' and grafana panel interpret it as CRITICAL.

request/response:
==============
SELECT last("value") FROM "haproxy_backend_status" WHERE "hostname" =~ /node-110/ AND time > now() - 5m GROUP BY time(1m)

{"results":[{"series":[{"name":"haproxy_backend_status","columns":["time","last"],"values":[[1457193600000,1],[1457193660000,1],[1457193720000,1],[1457193780000,1],[1457193840000,1],[1457193900000,null]]}]}]}

Solution
=======
add fill(previous) to avoid null results on ALL "Single stat" panel representing a service status

[Simon Pasquier (simon-pasquier)]

So it means that the "last" datapoint is missing? Maybe in that case it is more correct to say that the status is unknown?

[Swann Croiset (swann-w)]

So it seems that fill(none) makes the trick

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/401243

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/401243
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=768d468bf8010039fabe1203b27b50f7c763bbae
Submitter: Jenkins
Branch: master

commit 768d468bf8010039fabe1203b27b50f7c763bbae
Author: Swann Croiset <email address hidden>
Date: Wed Nov 23 13:52:01 2016 +0100

    Fix HAproxy dashboard

    * missing WHERE clause for the backend status
    * remove the GROUP_BY clause to avoid flapping

    Change-Id: Ifab23ad13e2621d5eb50a53753ebe63d44ae785a
    Related-bug: #1554993

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/1.0
Review: https://review.openstack.org/419944

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/419944
Committed: https://git.openstack.org/cgit/openstack/fuel-plugin-influxdb-grafana/commit/?id=b5e8156d8f80bf08ef876ecd4f1dd913063eae0b
Submitter: Jenkins
Branch: stable/1.0

commit b5e8156d8f80bf08ef876ecd4f1dd913063eae0b
Author: Swann Croiset <email address hidden>
Date: Wed Nov 23 13:52:01 2016 +0100

    Fix HAproxy dashboard

    * missing WHERE clause for the backend status
    * remove the GROUP_BY clause to avoid flapping

    Change-Id: Ifab23ad13e2621d5eb50a53753ebe63d44ae785a
    Related-bug: #1554993
    (cherry picked from commit 768d468bf8010039fabe1203b27b50f7c763bbae)