potential lost of messages by distributed notification agents

[gordon chung]

existing code will create queues from the cartesian product of the
set of pipelines and the set of agents. these queues are identified
by pipe_type-pipe_name-agent_id. the problem arises where if an agent
dies, any messages left on it's queue will be lost as agent_id's tend
to be unique.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/214813

[Rohit Jaiswal (rohit-jaiswal-3)]

Does this affect the notification agent HA in stable/kilo as well?

[gordon chung (chungg)]

nope. in kilo we didn't really distribute. we had one queue per pipeline... ie if there was 1 pipeline, 2 agents, only one of the agents would do the processing

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/214813
Committed: https://git.openstack.org/cgit/openstack/ceilometer/commit/?id=2e4b807131b4f21dbc4bcf34883592b738351168
Submitter: Jenkins
Branch: master

commit 2e4b807131b4f21dbc4bcf34883592b738351168
Author: gordon chung <email address hidden>
Date: Wed Aug 19 17:33:41 2015 -0400

    allow configurable pipeline partitioning

    existing code will create queues from the cartesian product of the
    set of pipelines and the set of agents. these queues are identified
    by pipe_type-pipe_name-agent_id. the problem arises where if an agent
    dies, any messages left on it's queue will be lost as agent_id's tend
    to be unique.

    this patch changes the logic so rather than using the number of agents
    as the set we build our cartesian product from, we use a set of numbers
    of a configurable size. by default, it will be a set of 10 so that queues
    are created from the product set of the set of pipelines and the set
    {0, 1, 2, 3, ... 9}. similarly, the pipelines queues will be identified
    by pipe_type-pipe_name-x, where x is value from numerical set. this
    way, if an agent dies, the queues are still known and can be
    redistributed and processed among the active agents.

    Change-Id: I35e3afdd2c9f97ab5bc6bb89531bd20308c5f8c0
    Closes-Bug: #1486754