I've also extracted the following traces from the logs https://gist.github.com/sigmavirus24/88e55420bfc7c9e089e7 (reproduced below). The last one is the same one that Bjoern pasted above. It seems to me that oslo.messaging should be catching some of these exceptions and attempting to do what is necessary to clean up the channel it was using for rabbit. Rabbit will not reap channels that are no longer in use or are effectively broken, that means it is the client's responsibility to do so.

There may be a case to be made that Kombu should be doing this, but at this point it would seem that oslo.messaging is well positioned to fix this for openstack and then work a patch into Kombu.

2014-12-05 18:29:48.642 1621 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on rabbit_host0:5672
2014-12-05 18:29:48.651 1621 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on rabbit_host0:5672
2014-12-09 17:24:58.445 1621 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: [Errno 104] Connection reset
by peer
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers
/impl_rabbit.py", line 655, in ensure
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit return method()
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers
/impl_rabbit.py", line 735, in _consume
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", l
ine 275, in drain_events
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.
py", line 91, in drain_events
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", li
ne 323, in drain_events
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", li
ne 526, in _close
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 554, in _x_close_ok
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 51))
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 56, in _send_method
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit self.channel_id, method_sig, args, content,
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 221, in write_method
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit write_frame(1, channel, payload)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 182, in write_frame
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce,
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 359, in sendall
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio.py", line 342, in send
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2014-12-09 17:24:58.445 1621 TRACE oslo.messaging._drivers.impl_rabbit
2014-12-09 17:24:58.447 1621 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds... 

---------------

2014-12-09 17:38:00.167 16249 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on rabbit_host1:5672
2014-12-09 17:38:00.178 16249 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on rabbit_host1:5672
2014-12-11 22:09:32.208 16249 WARNING neutron.openstack.common.loopingcall [-] task run outlasted interval by 5.797652 sec
2014-12-12 15:27:38.717 16249 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: Socket closed
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0]
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/oslo/messaging/_drivers/impl_rabbit.py", line 695, in _consume
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=1)
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 91, in drain_events
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 302, in drain_events
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout,
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 365, in _wait_multiple
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout)
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 336, in read_timeout
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method()
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit raise m
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit IOError: Socket closed
2014-12-12 15:27:38.717 16249 TRACE oslo.messaging._drivers.impl_rabbit
2014-12-12 15:27:38.720 16249 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server rabbit_host1:5672 closed the connection. Check login credentials: Socket closed 

---------------

2014-12-12 15:27:40.784 16249 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on rabbit_host1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-12-12 15:27:40.801 16249 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: [Errno 111] ECONNREFUSED
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 507, in __call__
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit self.revive(create_channel())
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 242, in channel
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit chan = self.transport.create_channel(self.connection)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 741, in connection
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit self._connection = self._establish_connection()
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/connection.py", line 696, in _establish_connection
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit conn = self.transport.establish_connection()
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 112, in establish_connection
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit conn = self.Connection(**opts)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 165, in __init__
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit self.transport = self.Transport(host, connect_timeout, ssl)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/connection.py", line 186, in Transport
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit return create_transport(host, connect_timeout, ssl)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 299, in create_transport
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit return TCPTransport(host, connect_timeout)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/local/lib/python2.7/dist-packages/amqp/transport.py", line 95, in __init__
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit raise socket.error(last_err)
2014-12-12 15:27:40.801 16249 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 111] ECONNREFUSED