Collector dispatches sample/ event multiple times to dispatcher when multiple dispatchers are configured

[Rohit Jaiswal]

The requeue_event_on_dispatcher_error is a good feature, but it may cause
data to be dispatched multiple times in below scenario-

When multiple dispatchers are enabled, if one dispatcher, say
database raise exception, then it will be requeued, but http dispatcher or file dispatcher may succeed, then the http dispatcher will handle that message twice, how could we avoid this scenario? does it depend on the http target to filter duplicate data?

[Rohit Jaiswal (rohit-jaiswal-3)]

It might be complex for a dispatcher to handle duplicate data. Is it possible for each dispatcher to requeue data on a different error queue specific to that dispatcher? So, each dispatcher will queue data on a dispatcher-specific error queue on a failure, instead of the collector requeue on metering queue. There are fixed number of dispatchers, so the number and size of these error queues is expected to be within bounds, assuming there is a limited number of failures.. This frees up the metering queue from the burden of failed data and also allows collector to continue accepting new data.

[ZhiQiang Fan (aji-zqfan)]

I think the right direction may be: dispatcher should handle duplicate data by its own.

[Rohit Jaiswal (rohit-jaiswal-3)]

I agree; i just think that having each dispatcher handle duplicate data in its own way is probably not the best solution. i mean i was thinking about an approach which would avoid data duplication with each new dispatcher having to do minimal work.

[Rohit Jaiswal (rohit-jaiswal-3)]

Here's what i was thinking -

1. When dispatcher fails to process a sample/event, it raises the exception to collector worker and queues the failed sample/event to a queue by the name - metering-<dispatcher>.error

2. Collector worker receives the exception and does not requeue the sample to metering queue. Collector calls a new method in each dispatcher to check for failed sample/event in their respective queue. Each dispatcher defines and knows its error queue (metering-<dispatcher>.error)

3. All known and configured dispatchers check their respective error queues for failed sample/event, but only the dispatchers that find something in their error queue attempt reprocessing of the failed sample/event by calling the respective storage layer repeatedly.

New dependency is of dispatcher with oslo.messaging

[gordon chung (chungg)]

so Rohit and i spoke on irc (http://eavesdrop.openstack.org/irclogs/%23openstack-ceilometer/%23openstack-ceilometer.2015-03-30.log)

this can be solved by not enabling multiple dispatchers but instead publishing to two different queues, and having two collectors (with a single dispatch target) listen to each one... i think because of that this bug is less important. also, i think it might actually be a bit more efficient to do it it this way than having a dispatcher manage its own set of queues.