Here is the log that shows the first couple of attempts to connect to rabbitmq.
2014-11-14 21:29:59.616 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:09.668 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:19.720 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:29.774 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:39.827 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:49.880 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:30:59.933 4284 WARNING nova.conductor.api [req-4457f21d-dde3-40f7-8825-b8c25fbaa098 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor?
2014-11-14 21:31:00.259 4284 WARNING nova.virt.libvirt.driver [req-de38aca0-7ac4-4b82-8465-ec97c66c6f8c None None] Cannot update service status on host: n141,since it is not registered.
2014-11-14 21:31:02.389 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672
2014-11-14 21:31:13.887 4284 WARNING nova.compute.manager [req-c2d8ca05-307a-473c-ba4d-c6b4211cad2c None None] Bandwidth usage not supported by hypervisor.
2014-11-14 21:34:17.591 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672
2014-11-14 21:34:32.917 4284 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to consume message from queue: [Errno 104] Connection reset by peer
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit Traceback (most recent call last):
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 622, in ensure
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return method(*args, **kwargs)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/impl_rabbit.py", line 702, in _consume
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return self.connection.drain_events(timeout=timeout)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/connection.py", line 139, in drain_events
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 223, in drain_events
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 56, in drain_events
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return self.wait_multi(self.channels.values(), timeout=timeout)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/kombu/transport/pyamqplib.py", line 81, in wait_multi
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit return amqp_method(channel, args)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 365, in _close
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit self._x_close_ok()
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/connection.py", line 384, in _x_close_ok
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit self._send_method((10, 61))
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 70, in _send_method
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit method_sig, args, content)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 233, in write_method
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit self.dest.write_frame(1, channel, payload)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/amqplib/client_0_8/transport.py", line 125, in write_frame
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit frame_type, channel, size, payload, 0xce))
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 359, in sendall
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit tail = self.send(data, flags)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.6/site-packages/eventlet/greenio.py", line 342, in send
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit total_sent += fd.send(data[total_sent:], flags)
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit error: [Errno 104] Connection reset by peer
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging._drivers.impl_rabbit
2014-11-14 21:34:32.919 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672
2014-11-14 21:34:32.919 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-11-14 21:34:33.920 4284 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.1.0.144:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds.
2014-11-14 21:34:34.920 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672
2014-11-14 21:34:34.921 4284 INFO oslo.messaging._drivers.impl_rabbit [-] Delaying reconnect for 1.0 seconds...
2014-11-14 21:34:35.921 4284 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 10.1.0.144:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds.
2014-11-14 21:34:37.523 4284 ERROR oslo.messaging._drivers.impl_rabbit [-] Failed to publish message to topic 'conductor': [Errno 104] Connection reset by peer
Here is the log that shows the first couple of attempts to connect to rabbitmq.
2014-11-14 21:29:59.616 4284 WARNING nova.conductor.api [req-4457f21d- dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? dde3-40f7- 8825-b8c25fbaa0 98 None None] Timed out waiting for nova-conductor. Is it running? Or did this service start before nova-conductor? libvirt. driver [req-de38aca0- 7ac4-4b82- 8465-ec97c66c6f 8c None None] Cannot update service status on host: n141,since it is not registered. _drivers. impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672 manager [req-c2d8ca05- 307a-473c- ba4d-c6b4211cad 2c None None] Bandwidth usage not supported by hypervisor. _drivers. impl_rabbit [-] Connected to AMQP server on 10.1.0.144:5672 _drivers. impl_rabbit [-] Failed to consume message from queue: [Errno 104] Connection reset by peer _drivers. impl_rabbit Traceback (most recent call last): _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 622, in ensure _drivers. impl_rabbit return method(*args, **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ oslo/messaging/ _drivers/ impl_rabbit. py", line 702, in _consume _drivers. impl_rabbit return self.connection .drain_ events( timeout= timeout) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/connectio n.py", line 139, in drain_events _drivers. impl_rabbit return self.transport. drain_events( self.connection , **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/transport /pyamqplib. py", line 223, in drain_events _drivers. impl_rabbit return connection. drain_events( **kwargs) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/transport /pyamqplib. py", line 56, in drain_events _drivers. impl_rabbit return self.wait_ multi(self. channels. values( ), timeout=timeout) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ kombu/transport /pyamqplib. py", line 81, in wait_multi _drivers. impl_rabbit return amqp_method( channel, args) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqplib/ client_ 0_8/connection. py", line 365, in _close _drivers. impl_rabbit self._x_close_ok() _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqplib/ client_ 0_8/connection. py", line 384, in _x_close_ok _drivers. impl_rabbit self._send_ method( (10, 61)) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 70, in _send_method _drivers. impl_rabbit method_sig, args, content) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqplib/ client_ 0_8/method_ framing. py", line 233, in write_method _drivers. impl_rabbit self.dest. write_frame( 1, channel, payload) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ amqplib/ client_ 0_8/transport. py", line 125, in write_frame _drivers. impl_rabbit frame_type, channel, size, payload, 0xce)) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ eventlet/ greenio. py", line 359, in sendall _drivers. impl_rabbit tail = self.send(data, flags) _drivers. impl_rabbit File "/usr/lib/ python2. 6/site- packages/ eventlet/ greenio. py", line 342, in send _drivers. impl_rabbit total_sent += fd.send( data[total_ sent:], flags) _drivers. impl_rabbit error: [Errno 104] Connection reset by peer _drivers. impl_rabbit _drivers. impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672 _drivers. impl_rabbit [-] Delaying reconnect for 1.0 seconds... _drivers. impl_rabbit [-] AMQP server on 10.1.0.144:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. _drivers. impl_rabbit [-] Reconnecting to AMQP server on 10.1.0.144:5672 _drivers. impl_rabbit [-] Delaying reconnect for 1.0 seconds... _drivers. impl_rabbit [-] AMQP server on 10.1.0.144:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 3 seconds. _drivers. impl_rabbit [-] Failed to publish message to topic 'conductor': [Errno 104] Connection reset by peer
2014-11-14 21:30:09.668 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:30:19.720 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:30:29.774 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:30:39.827 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:30:49.880 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:30:59.933 4284 WARNING nova.conductor.api [req-4457f21d-
2014-11-14 21:31:00.259 4284 WARNING nova.virt.
2014-11-14 21:31:02.389 4284 INFO oslo.messaging.
2014-11-14 21:31:13.887 4284 WARNING nova.compute.
2014-11-14 21:34:17.591 4284 INFO oslo.messaging.
2014-11-14 21:34:32.917 4284 ERROR oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.917 4284 TRACE oslo.messaging.
2014-11-14 21:34:32.919 4284 INFO oslo.messaging.
2014-11-14 21:34:32.919 4284 INFO oslo.messaging.
2014-11-14 21:34:33.920 4284 ERROR oslo.messaging.
2014-11-14 21:34:34.920 4284 INFO oslo.messaging.
2014-11-14 21:34:34.921 4284 INFO oslo.messaging.
2014-11-14 21:34:35.921 4284 ERROR oslo.messaging.
2014-11-14 21:34:37.523 4284 ERROR oslo.messaging.