aggregation: issue when one of the resource is no more updated

[Mehdi Abaakouk]

Hi,

When across metric aggregation are used by heat and aodh the following happen:

* heat creates a autoscaling group with 3 servers
* gnocchi got 3 three new resources for theses servers with their measurements
* then the autoscaling stuff decide to remove one server
* metrics of one of the gnocchi resources is no more updated with new measurements
* gnocchi "across metric aggregation" endpoint always returns:

    <h1>400 Bad Request</h1>
    The server could not comply with the request since it is either malformed or otherwise incorrect.<br /><br />
     One of the metric to aggregated doesn't have matching granularity

The problem is that we can't really known the reason of why the measurements that doesn't come anymore.
Perhaps it just a temporary issue and they are nothing todo, perhaps the resource doesn't exists anymore outside of
gnocchi and will be not updated anymore.

Cheers,

[Mehdi Abaakouk (sileht)]

A smart solution would be to set "ended_at" on the resource, when the instance is deleted and filter out instances with this flags set, so the aggregation result will be always correct.

But we don't have such thing in the ceilometer dispatcher yet, so the proposed solution will be:
* Implements the "ended_at" thing for the futur
* Set 'percent_of_overlap=0' temporary in aodh for gnocchi alarms against "aggregated metrics across resources".

[Mehdi Abaakouk (sileht)]

Aodh workaround: https://review.openstack.org/#/c/209109/

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/209109
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=75b2c47db7f3c092b8bfa47de8e49036041762cc
Submitter: Jenkins
Branch: master

commit 75b2c47db7f3c092b8bfa47de8e49036041762cc
Author: Mehdi Abaakouk <email address hidden>
Date: Tue Aug 4 16:49:21 2015 +0200

    gnocchi: percent_of_overlap=0 for agg. alarms

    In case of a heat autoscaling stack decide to delete an instance,
    the gnocchi metrics associated to this instance will be no more
    updated and when the alarm will ask for the aggregation, gnocchi
    will raise a 'No overlap' exception.
    So temporary set 'percent_of_overlap' to 0 to disable the
    gnocchi checks about missing points.

    And adds some docs about the importance of filter out resource
    with ended_at != 'null'.

    For more detail see:
      https://bugs.launchpad.net/gnocchi/+bug/1479429

    Related bug: #1479429

    Change-Id: I528d21c048d84b9552febd08f787812c668983ab

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/338901

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/338896
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=b11d9368a140883ef1ff3924f22377f4006c3a7b
Submitter: Jenkins
Branch: master

commit b11d9368a140883ef1ff3924f22377f4006c3a7b
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jul 7 13:30:29 2016 +0200

    gnocchi: always set needed_overlap for aggregation

    gnocchi_aggregation_by_resources_threshold set needed_overlap
    to workaround #1479429, we should do the same for
    gnocchi_aggregation_by_metrics_threshold.

    This change does it.

    Related-bug: #1479429
    Change-Id: I283200e23d69dbd174f990722711d6b538fe42e3

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/338901
Committed: https://git.openstack.org/cgit/openstack/aodh/commit/?id=5c98334eed12e009b5e4c83bd65f4a8acc69f260
Submitter: Jenkins
Branch: stable/mitaka

commit 5c98334eed12e009b5e4c83bd65f4a8acc69f260
Author: Mehdi Abaakouk <email address hidden>
Date: Thu Jul 7 13:30:29 2016 +0200

    gnocchi: always set needed_overlap for aggregation

    gnocchi_aggregation_by_resources_threshold set needed_overlap
    to workaround #1479429, we should do the same for
    gnocchi_aggregation_by_metrics_threshold.

    This change does it.

    Related-bug: #1479429
    Change-Id: I283200e23d69dbd174f990722711d6b538fe42e3
    (cherry picked from commit cd7d7b27eb04f58c9013e394c68b3c4a1ea7c419)