should raise expception when rabbit broker has blocked the connection

[Sungjin Yook]

Issue statement
=============
When rabbitMQ broker has blocked the connection, oslo.messaging._drivers.impl_rabbit logs the 'ERROR', but it does not raise exception. Because of this whoever cast() or call() via oslo.messaging._drivers.impl_rabbit hangs.

For example, from barbican side, when it casts message via oslo.messaging._drivers.impl_rabbit, it hangs because oslo.messaging._drivers.impl_rabbit never raise proper expception. Thus waiting forever even in case that the message was not delivered successfully to rabbitMQ.

The details of error on barbican side shown below:
========================================
2015-11-14 00:51:14.897 ERROR oslo.messaging._drivers.impl_rabbit [req-5e64a213-96d9-4692-b82c-b6e7146b7da0 fb266c81338849b8bcde547d3975f2a5 fe97b85c488d4c47afe7e59df8a2c602] The broker has blocked the connection: low on disk
<== Barbican hangs right after this message.

Suggested solution
================
In oslo_messaging/_drivers/impl_rabbit.py,
    def _on_connection_blocked(reason):
        LOG.error(_LE("The broker has blocked the connection: %s"), reason)

After logging ERROR for rabbit broker blocking connection, the corresponding ERROR case exception should be raised to unblock olso.messaing user.

[Mehdi Abaakouk (sileht)]

Blocked connection uses same behavior as when rabbitmq is unreachable.

You have to solve the real issue to unblock your messages.

[Sungjin Yook (sungyook)]

In this case rabbitMQ let us know that it's intentionally blocking any connection due to 'low on disk'. barbican would wait forever for this failed msg delivery if no exception raised from oslo.messaging.

[Sungjin Yook (sungyook)]

@Mehdi I am glad that we are having this discussion and thanks for your insight. Would you have suggestion other than resolving rabbitMQ issue to unblock ? For barbican, it would be best if we can return error to user immediately with proper msg that rabbitMQ blocked connection. Waiting until rabbitMQ issue being resolved is not desirable.
I really appreciate your input on this.

[OpenStack Infra (hudson-openstack)]

Change abandoned by ChangBo Guo(gcb) (<email address hidden>) on branch: master
Review: https://review.openstack.org/245969
Reason: Please restore it if you want to continue working on this.