ipc-sample queues keep growing

[Christian Zunker]

We are using ceilometer 10.0.2.dev2 in an OpenStack Queens installation.

After some time some of the ceilometer_ipc-sample-*.sample queues grow unexpectedly. Normally these queues hold up to 100 messages until they are consumed. But after some time without restart of ceilometer-agent-notification, some queues (4-5) grow up to multiple thousand messages.

When we than restart all ceilometer-agent-notification these queues get consumed again (s. drop of messages on the rifht hand side (~5:05) of attached screenshot).

We have a HA setup with 3 controllers. These are our settings:
[notification]
batch_size = 500
messaging_urls = ...
store_events = False
workers = 16
workload_partitioning = True
pipeline_processing_queues = 48