Fuel for OpenStack

  1. Bug #1463433
  2. Comment #19

Comment 19 for bug 1463433

Revision history for this message
Bogdan Dobrelya (bogdando) wrote : Re: [shaker] test failing due to multiple "Timed out waiting for reply to ID" events logged by Oslo.messaging after rabbitmq recovered from partitioning and kept running with AMQP publish got blocked because virt memory got exhausted at rabbit node

Folks, I tested how exactly blocked publish works when a rabbit node exceeds its high memory watermark.
Here is the scripts: http://pastebin.com/49JisHRP , If you want to give a try, make sure you provided correct IP addresses and nova creds. And the sample generator itself https://github.com/bogdando/ceilometer/raw/rmq_bench/tools/sample-generator.py ,
it requires pika and python-ceilometer installed. The output of the test was http://pastebin.com/EgYtzYuY

 As you can see from test results, consume never got blocked on raised RabbitMQ memory alert, only publish is blocked. So this issue with the swap increasing in uncontrolled way by beam.smp process is likely related to OpenStack apps keep pushing on rabbit cluster when it has declared blocked publish. The pressure on the memory may be done, for example, by new connections opening.

The recent patch should give us memory grow dynamics for queues, connections etc.