AMQP connections to RabbitMQ nodes should be load balanced

[Bogdan Dobrelya]

In the scope of related bug https://bugs.launchpad.net/bugs/1421611, the rabbitmq single connection point was removed from the HAProxy as Oslo.messaging started supporting multiple AMQP hosts in configuration.

But we can imagine the following scenario which may break the rabbitmq cluster apart under load:
1) Some node A fails and all its AMQP connections switched to another nodes B, C so the load for them will increase by 1/3 each.
2) When node A recovered, it would be running with no load until new connections established. But there is no load balancing and we can expect new connections will land on the nodes A, B, C by 1/3 each again.
3) If the node A continues to fail and recover, the load to the nodes B and C will grow badly.

So, the new AMQP sessions should be accepted by nodes based on the least connections criterion.

[Bogdan Dobrelya (bogdando)]

Added MOS project for the case we decide to make LB at the app level.

[Andrew Woodward (xarses)]

We had removed the LB not because of the support of rabbit_hosts increased, but because we found issues with stale connections during failure and fail-over situations. Now that we have connection heatbeats and the lb is in a network namespace, this may be moot. As an extra measure please ensure that this undergoes extensive regression testing. We should create some automated tests to simplify this down the road

[Davanum Srinivas (DIMS) (dims-v)]

upstream oslo.messaging team and ops folks do *NOT* recommend using HAProxy. rabbit_hosts is what everyone is using and testing with. (Example - http://markmail.org/message/dzyokzmtu2khdish)

[Bogdan Dobrelya (bogdando)]

thank you for update, marking this bug as won't fix