tried it again.. restarted rabbitmq-server.. tried to spawn 2 new instances a few seconds after the start of rabbitmq-server: ---snip--- ares:~ # euca-describe-instances RESERVATION r-n6jwk5yj testing default INSTANCE i-00000107 ami-2a1izx00 scheduling foobar (testing, None) 0 m1.tiny 2011-02-23T13:52:56Z unknown zone INSTANCE i-00000108 ami-2a1izx00 scheduling foobar (testing, None) 1 m1.tiny 2011-02-23T13:52:56Z unknown zone ---snap--- ---snip--- (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/nova/rpc.py", line 120, in fetch (nova.rpc): TRACE: super(Consumer, self).fetch(no_ack, auto_ack, enable_callbacks) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/messaging.py", line 307, in fetch (nova.rpc): TRACE: message = self.backend.get(self.queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/backends/pyamqplib.py", line 277, in get (nova.rpc): TRACE: raw_message = self.channel.basic_get(queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/channel.py", line 2032, in basic_get (nova.rpc): TRACE: (60, 72), # Channel.basic_get_empty (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 89, in wait (nova.rpc): TRACE: self.channel_id, allowed_methods) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 218, in _wait_method (nova.rpc): TRACE: self.wait() (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 105, in wait (nova.rpc): TRACE: return amqp_method(self, args) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 367, in _close (nova.rpc): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id)) (nova.rpc): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '') (nova.rpc): TRACE: 2011-02-23 14:52:46,720 ERROR nova.rpc [-] Failed to fetch message from queue (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/nova/rpc.py", line 120, in fetch (nova.rpc): TRACE: super(Consumer, self).fetch(no_ack, auto_ack, enable_callbacks) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/messaging.py", line 307, in fetch (nova.rpc): TRACE: message = self.backend.get(self.queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/backends/pyamqplib.py", line 277, in get (nova.rpc): TRACE: raw_message = self.channel.basic_get(queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/channel.py", line 2032, in basic_get (nova.rpc): TRACE: (60, 72), # Channel.basic_get_empty (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 89, in wait (nova.rpc): TRACE: self.channel_id, allowed_methods) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 218, in _wait_method (nova.rpc): TRACE: self.wait() (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 105, in wait (nova.rpc): TRACE: return amqp_method(self, args) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 365, in _close (nova.rpc): TRACE: self._x_close_ok() (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 384, in _x_close_ok (nova.rpc): TRACE: self._send_method((10, 61)) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 70, in _send_method (nova.rpc): TRACE: method_sig, args, content) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/method_framing.py", line 233, in write_method (nova.rpc): TRACE: self.dest.write_frame(1, channel, payload) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/transport.py", line 125, in write_frame (nova.rpc): TRACE: frame_type, channel, size, payload, 0xce)) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/eventlet/greenio.py", line 283, in sendall (nova.rpc): TRACE: tail = self.send(data, flags) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/eventlet/greenio.py", line 269, in send (nova.rpc): TRACE: total_sent += fd.send(data[total_sent:], flags) (nova.rpc): TRACE: error: [Errno 32] Broken pipe (nova.rpc): TRACE: ---snap--- tried it again a few minutes after starting rabbitmq-server i can still not spawn new instances (but i got no new errors in the logfiles) ---snip--- ares:~ # euca-describe-instances RESERVATION r-n6jwk5yj testing default INSTANCE i-00000107 ami-2a1izx00 scheduling foobar (testing, None) 0 m1.tiny 2011-02-23T13:52:56Z unknown zone INSTANCE i-00000108 ami-2a1izx00 scheduling foobar (testing, None) 1 m1.tiny 2011-02-23T13:52:56Z unknown zone RESERVATION r-4aulwrb9 testing default INSTANCE i-00000109 ami-2a1izx00 scheduling foobar (testing, None) 0 m1.tiny 2011-02-23T13:57:46Z unknown zone INSTANCE i-0000010a ami-2a1izx00 scheduling foobar (testing, None) 1 m1.tiny 2011-02-23T13:57:46Z unknown zone ---snap--- now i restarted all nova services and restarted rabbitmq-server afterwards again after a few minutes i tried to spawn 2 instances... still not working... got the following error in the logs: ---snip--- 2011-02-23 15:02:12,761 ERROR nova.rpc [-] Failed to fetch message from queue (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/nova/rpc.py", line 120, in fetch (nova.rpc): TRACE: super(Consumer, self).fetch(no_ack, auto_ack, enable_callbacks) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/messaging.py", line 307, in fetch (nova.rpc): TRACE: message = self.backend.get(self.queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/backends/pyamqplib.py", line 277, in get (nova.rpc): TRACE: raw_message = self.channel.basic_get(queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/channel.py", line 2032, in basic_get (nova.rpc): TRACE: (60, 72), # Channel.basic_get_empty (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 89, in wait (nova.rpc): TRACE: self.channel_id, allowed_methods) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 218, in _wait_method (nova.rpc): TRACE: self.wait() (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 105, in wait (nova.rpc): TRACE: return amqp_method(self, args) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 367, in _close (nova.rpc): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id)) (nova.rpc): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '') (nova.rpc): TRACE: 2011-02-23 15:02:12,768 ERROR nova.rpc [-] Failed to fetch message from queue (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/nova/rpc.py", line 120, in fetch (nova.rpc): TRACE: super(Consumer, self).fetch(no_ack, auto_ack, enable_callbacks) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/messaging.py", line 307, in fetch (nova.rpc): TRACE: message = self.backend.get(self.queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/carrot/backends/pyamqplib.py", line 277, in get (nova.rpc): TRACE: raw_message = self.channel.basic_get(queue, no_ack=no_ack) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/channel.py", line 2032, in basic_get (nova.rpc): TRACE: (60, 72), # Channel.basic_get_empty (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 89, in wait (nova.rpc): TRACE: self.channel_id, allowed_methods) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 218, in _wait_method (nova.rpc): TRACE: self.wait() (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/abstract_channel.py", line 105, in wait (nova.rpc): TRACE: return amqp_method(self, args) (nova.rpc): TRACE: File "/usr/lib64/python2.6/site-packages/amqplib/client_0_8/connection.py", line 367, in _close (nova.rpc): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id)) (nova.rpc): TRACE: AMQPConnectionException: (320, u"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'", (0, 0), '') (nova.rpc): TRACE: ---snap---