If RabbitMQ fails, crashes, or is restarted, the services connected to RabbitMQ don't attempt to re-establish their connections. All services currently need to be restarted to reconnect and recreate the queues, otherwise they just sit in place running. Services should attempt to reconnect to RabbitMQ in that situation and try and reestablish their queues.
If RabbitMQ is shutdown, services will receive this exception:
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 316, in wait
(nova.rpc): TRACE: it.next()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 287, in consume
(nova.rpc): TRACE: self.channel.wait()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.rpc): TRACE: self.channel_id, allowed_methods)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 218, in _wait_method
(nova.rpc): TRACE: self.wait()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 105, in wait
(nova.rpc): TRACE: return amqp_method(self, args)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 367, in _close
(nova.rpc): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))
(nova.rpc): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '')
If RabbitMQ is killed, services will receive this exception:
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 316, in wait
(nova.rpc): TRACE: it.next()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 287, in consume
(nova.rpc): TRACE: self.channel.wait()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/abstract_channel.py", line 89, in wait
(nova.rpc): TRACE: self.channel_id, allowed_methods)
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 198, in _wait_method
(nova.rpc): TRACE: self.method_reader.read_method()
(nova.rpc): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/method_framing.py", line 215, in read_method
(nova.rpc): TRACE: raise m
(nova.rpc): TRACE: IOError: Socket closed
(Running 1156 of Nova)
This was supposed to be addressed in bug 718869 ?