Collector continuously re-queues sample when dispatcher reports persistent error when requeue is enabled

[Rohit Jaiswal]

When requeue_sample_on_dispatcher_error is enabled in ceilometer.conf, the collector will try to requeue the sample if it gets an error from the dispatcher.

When the sample gets requeued, it will get picked up again and dispatched to the storage layer. If the same underlying error condition prevails, the error will be raised back to the collector and the message will be requeued again. This cyclical process will continue until the error is gone(in which case the sample is not requeued again) or the collector/ RMQ are restarted.

In this scenario, when there is an persistent error condition, frequent retrying puts extra load on storage db and the messaging layer (rabbit), wasting collector CPU cycles since the message and potentially more samples keep getting requeued and not cleared from the queue.

It does not make sense to keep retrying continuously in case of a persistent error condition.

There should be a configurable upper limit to cap the number of requeues of samples/events by Collector in case of dispatcher error.

eg. requeue_sample_on_dispatcher_error_max_retries

[gordon chung (chungg)]

what's the proposed solution here? i get the feeling this has multiple different solutions all with pros/cons.

[Rohit Jaiswal (rohit-jaiswal-3)]

Using a new configuration param - requeue_max_retries (in ceilometer.conf under collector section) which will control how many times the collector tries to requeue sample/event in case of a repetitive failure. Once the retries max out, the sample will be dropped or can be put on a metering.error queue (maybe a durable queue), so that we dont entirely lose it and free up the collector from having to retry it. The assumption is the percentage of these error cases is low, so we dont end up with a bloated error queue.

By default, the requeue_max_retries will have a value of -1, which indicates infinite retries, the current behavior.

[gordon chung (chungg)]

so i guess why i asked was because for the above case, how would you know how many times you've retried a sample/event? are we marking the datapoints somehow? if you requeue it, there's really no way to know how many times you've tried something unless you keep a cache...and then you'd need a global cache to sync across multiple workers...

[Rohit Jaiswal (rohit-jaiswal-3)]

 I agree that a global cache will be needed to synchronize access and update the retry count by multiple collector worker processes. Thanks for describing this. I think we can use the distributed locking in Tooz with redis to cache the retry count. Workers will acquire the lock to update the retry count, so this might impact performance, but this will only be when requeueing is enabled and only needed to update the cache, so should be insignificant. I think this sounds more like a new feature than a bug. Let me know what you think.

[Rohit Jaiswal (rohit-jaiswal-3)]

Thinking a bit more, we would also need a way to identify each sample that gets requeued, so as to maintain its retry count in the cache. This could be done in redis using key/value pairs. So we could use the unique sample id or hash a sample to generate a unique id which we use then use to look up the retry count in redis. So we can maintain multiple sample requeue retry counts in cache and use tooz to synchronize access to this struct by multiple workers which potentially may be retrying the same or different sample(s).

[gordon chung (chungg)]

i'm not sure sample_id is unique -- the hash technique sounds good though... or whatever we're setting as unique id in sql.

tooz could work? we could also look at dogpile.cache which has some expiration capabilities.

[Rohit Jaiswal (rohit-jaiswal-3)]

dogpile.cache seems like a good fit, keystone already uses it. I dont think ceilometer has a cache layer, that said, i think this a new feature to ceilometer (collector) and the new requirement of limiting the requeuing. Does this deserve a blueprint?

[gordon chung (chungg)]

n/a. storage is dead. long live gnocchi (or whatever proprietary/open solution you like)