Comment 9 for bug 1551667

I was also abble to reproduce this problem with oslo.messaging 1.7.1 and 4.0.0 (I hadn't checked 4.1.0+ as there is too many dependency conflicts to test it with liberty).

I tried to examine this more closely and the problem is in fact not in the part of AMQP listener but in dispatcher (for version 2.5.0 notify/dispatcher.py), where there is no check on how many incoming messages are being processed.

This is what happens:
The message is obtained from listener and is processed by endpoint (in my case both mongo and gnocchi) executor, but since we use eventlet (and there is usually lot of space to interrupt processing of current "thread" in executors), the processing is given to another "thread" which tries to obtain yet another message. This results in a sequence get message - start processing message - get message - start processing message - ...all other messages in queue... - finish processing - aknowledge - finish processing - aknowledge; which obviously fills RAM when there is too many messages in the queue.

I managed to somehow fix this by adding simple counter into NotificationDispatcher.__call__ that keeps calling eventlet.sleep when there is already too many messages being processed.

I'm not sure wheter this bug is still present in current master - I haven't noticed anything in code that would deal with this situation, but the code in master has evolved a lot from 2.5.0 and I could easily overlooked something.