Pollsters now send notifications without doing transforms

[OpenStack Infra]

https://review.openstack.org/198865
commit 5b9e5d8e7a45badc4741aff241f887adcf2312ab
Author: Chris Dent <email address hidden>
Date: Mon Jul 6 19:17:55 2015 +0000

    Pollsters now send notifications without doing transforms

    This makes the polling agents not use pipelines. Instead it simply sends
    notifications for the notification agent to pick up and transform if
    the pipeline.yaml says it should.

    Inside the AgentManager and the PollingTask the data representation
    is adjusted somewhat. Rather than making a single task for any given
    interval, we make a single task for any name in the "sources" list.
    This ought to mean (given that the interval is the same across
    various sources in the default config) that we will get some I/Ox
    interleaving.

    At the moment all samples gathered by one pollng task are sent as an
    individual notification. This is being done to minimize the apparent
    surface area of this change. The expected long term change is for
    single samples to be sent so as to increase granularity and
    I/O interleaving.

    The unit tests have been updated to reflect the new data
    representation. The agent tests are fairly strongly oriented towards
    testing that discovery and resource handling behave correctly. Some
    additions have been made to make sure that samples traverse a fake
    messaging bus as expected. Coverage of the ceilometer/agent/base has
    increased from 98 to 99%. Additional functional testing should be
    implemented when we have established the infrastructure for such
    things.

    Implements blueprint pollsters-no-transform

    DocImpact

    Change-Id: I25c22077e80509799713571dfd79c87fe21c8677

[Lana (loquacity)]

No openstack-manuals impact.

[Gauvain Pocentek (gpocentek)]

Setting the status back to NEW after discussion with Chris on IRC.

[Chris Dent (cdent)]

There are several pending changes to the pollsters and notification agents that will impact the docs. These changes will probably best come in as a single change that they are fairly intertwingled.

For sake of future reference the impact of this change is:

* The polling agents (ceilometer-agent-compute, ceilometer-agent-central, ceilometer-agent-ipmi OR ceilometer-polling (the new all-in-one version of polling)) will be lighter weight because they move pipeline transformation processing to the notification agents which may or may not be located on the same machine. This should make it easier to have more horizontally scaled polling agents on more machines
* The notification agent now will be doing more processing because in addition to the handling of notifications it was already handling it will now be handling notifications from the polling agents and doing any required transformations on those. This may cause increased load for those agents. That load can be ameliorated by running more notification agents on more machines: They can horizontally scale and use group coordination to distribute tasks.

There is a related pending change which will make notification agent group coordination more distributed: https://review.openstack.org/#/c/203818/

[Alexandra Settle (alexandra-settle)]

Chris - how relevant is this bug report at the moment? Could you update us on the impact?

[Launchpad Janitor (janitor)]

[Expired for openstack-manuals because there has been no activity for 60 days.]