I tried again on freshly reinstalled OS and the behavior is repeatable. I also saw one case where there was a slightly different AMPQ error that happened during instance termination:
headline: AMQPConnectionException: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open')
2011-09-21 11:33:05,739 DEBUG nova.utils [-] Attempting to grab semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1094) inner /usr/lib/pymodules/python2.7/nova/utils.py:672 2011-09-21 11:33:05,746 ERROR nova.rpc [-] Exception during message handling (nova.rpc): TRACE: Traceback (most recent call last): (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 626, in _process_data (nova.rpc): TRACE: ctxt.reply(rval, None) (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply (nova.rpc): TRACE: msg_reply(self.msg_id, *args, **kwargs) (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 781, in msg_reply (nova.rpc): TRACE: conn.direct_send(msg_id, msg) (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__ (nova.rpc): TRACE: self._done() (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done (nova.rpc): TRACE: self.connection.reset() (nova.rpc): TRACE: File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 383, in reset (nova.rpc): TRACE: self.channel = self.connection.channel() (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 99, in channel (nova.rpc): TRACE: return self.transport.create_channel(self.connection) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 197, in create_channel (nova.rpc): TRACE: return connection.channel() (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 130, in channel (nova.rpc): TRACE: return Channel(self, channel_id) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in __init__ (nova.rpc): TRACE: self._x_open() (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 471, in _x_open (nova.rpc): TRACE: (20, 11), # Channel.open_ok (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 95, in wait (nova.rpc): TRACE: self.channel_id, allowed_methods) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 231, in _wait_method (nova.rpc): TRACE: self.wait() (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait (nova.rpc): TRACE: return self.dispatch_method(method_sig, args, content) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method (nova.rpc): TRACE: return amqp_method(self, args) (nova.rpc): TRACE: File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 380, in _close (nova.rpc): TRACE: raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id)) (nova.rpc): TRACE: AMQPConnectionException: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') (nova.rpc): TRACE: 2011-09-21 11:33:05,780 ERROR nova.rpc [-] Returning exception (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') to caller 2011-09-21 11:33:05,780 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 626, in _process_data\n ctxt.reply(rval, None)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 673, in reply\n msg_reply(self.msg_id, *args, **kwargs)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_\ kombu.py", line 781, in msg_reply\n conn.direct_send(msg_id, msg)\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 562, in __exit__\n self._done()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 547, in _done\n self.connection.reset()\n', ' File "/usr/lib/pymodules/python2.7/nova/rpc/impl_kombu.py", line 383, in reset\n self.ch\ annel = self.connection.channel()\n', ' File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 99, in channel\n return self.transport.create_channel(self.connection)\n', ' File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 197, in create_channel\n return connection.channel()\n', ' File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqpli\ b.py", line 130, in channel\n return Channel(self, channel_id)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 82, in __init__\n self._x_open()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/channel.py", line 471, in _x_open\n (20, 11), # Channel.open_ok\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abst\ ract_channel.py", line 95, in wait\n self.channel_id, allowed_methods)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 231, in _wait_method\n self.wait()\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait\n return self.dispatch_method(method_sig, args, content)\n', ' File "/usr/lib/pytho\ n2.7/dist-packages/amqplib/client_0_8/abstract_channel.py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/connection.py", line 380, in _close\n raise AMQPConnectionException(reply_code, reply_text, (class_id, method_id))\n', 'AMQPConnectionException: (503, u"COMMAND_INVALID - second \'channel.open\' s\ een", (20, 10), \'Channel.open\')\n']
I tried again on freshly reinstalled OS and the behavior is repeatable. I also saw one case where there was a slightly different AMPQ error that happened during instance termination:
headline: AMQPConnectionE xception: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open')
2011-09-21 11:33:05,739 DEBUG nova.utils [-] Attempting to grab semaphore "get_dhcp" for method "_get_dhcp_ip"... from (pid=1094) inner /usr/lib/ pymodules/ python2. 7/nova/ utils.py: 672 pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 626, in _process_data pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 673, in reply self.msg_ id, *args, **kwargs) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 781, in msg_reply send(msg_ id, msg) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 562, in __exit__ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 547, in _done .reset( ) pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 383, in reset .channel( ) python2. 7/dist- packages/ kombu/connectio n.py", line 99, in channel create_ channel( self.connection ) python2. 7/dist- packages/ kombu/transport /pyamqplib. py", line 197, in create_channel channel( ) python2. 7/dist- packages/ kombu/transport /pyamqplib. py", line 130, in channel python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 82, in __init__ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 471, in _x_open python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 95, in wait python2. 7/dist- packages/ amqplib/ client_ 0_8/connection. py", line 231, in _wait_method python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait method( method_ sig, args, content) python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method python2. 7/dist- packages/ amqplib/ client_ 0_8/connection. py", line 380, in _close xception( reply_code, reply_text, (class_id, method_id)) xception: (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 626, in _process_data\n ctxt.reply(rval, None)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 673, in reply\n msg_reply( self.msg_ id, *args, **kwargs)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ \ send(msg_ id, msg)\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 562, in __exit__\n self._done()\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 547, in _done\n self.connection .reset( )\n', ' File "/usr/lib/ pymodules/ python2. 7/nova/ rpc/impl_ kombu.py" , line 383, in reset\n self.ch\ .channel( )\n', ' File "/usr/lib/ python2. 7/dist- packages/ kombu/connectio n.py", line 99, in channel\n return self.transport. create_ channel( self.connection )\n', ' File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqplib. py", line 197, in create_channel\n return connection. channel( )\n', ' File "/usr/lib/ python2. 7/dist- packages/ kombu/transport /pyamqpli\ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 82, in __init__\n self._x_open()\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/channel. py", line 471, in _x_open\n (20, 11), # Channel.open_ok\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abst\ methods) \n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/connection. py", line 231, in _wait_method\n self.wait()\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 97, in wait\n return self.dispatch_ method( method_ sig, args, content)\n', ' File "/usr/lib/pytho\ packages/ amqplib/ client_ 0_8/abstract_ channel. py", line 115, in dispatch_method\n return amqp_method(self, args)\n', ' File "/usr/lib/ python2. 7/dist- packages/ amqplib/ client_ 0_8/connection. py", line 380, in _close\n raise AMQPConnectionE xception( reply_code, reply_text, (class_id, method_id))\n', 'AMQPConnection Exception: (503, u"COMMAND_INVALID - second \'channel.open\' s\ open\') \n']
2011-09-21 11:33:05,746 ERROR nova.rpc [-] Exception during message handling
(nova.rpc): TRACE: Traceback (most recent call last):
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: ctxt.reply(rval, None)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: msg_reply(
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: conn.direct_
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self._done()
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self.connection
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self.channel = self.connection
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return self.transport.
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return connection.
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return Channel(self, channel_id)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self._x_open()
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: (20, 11), # Channel.open_ok
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self.channel_id, allowed_methods)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: self.wait()
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return self.dispatch_
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: return amqp_method(self, args)
(nova.rpc): TRACE: File "/usr/lib/
(nova.rpc): TRACE: raise AMQPConnectionE
(nova.rpc): TRACE: AMQPConnectionE
(nova.rpc): TRACE:
2011-09-21 11:33:05,780 ERROR nova.rpc [-] Returning exception (503, u"COMMAND_INVALID - second 'channel.open' seen", (20, 10), 'Channel.open') to caller
2011-09-21 11:33:05,780 ERROR nova.rpc [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/
kombu.py", line 781, in msg_reply\n conn.direct_
annel = self.connection
b.py", line 130, in channel\n return Channel(self, channel_id)\n', ' File "/usr/lib/
ract_channel.py", line 95, in wait\n self.channel_id, allowed_
n2.7/dist-
een", (20, 10), \'Channel.