nova-consoleauth issue with rabbitmq when trying to view console

[Jacob Godin]

Running into a problem when trying to open a noVNC console session. Screen hangs on "Starting VNC handshake", and eventually times out with a 1006 error.

I'm running the latest from the Grizzly Ubuntu Testing PPA.

nova-consoleauth.log shows:
2013-02-07 15:48:34.705 AUDIT nova.consoleauth.manager [req-d7094194-b6c4-4784-b265-a401dbffd32e None None] Checking Token: 93d09852-3b0b-4999-a9ca-79c75ab1ed83, True)
2013-02-07 15:48:34.709 ERROR nova.openstack.common.rpc.amqp [req-d7094194-b6c4-4784-b265-a401dbffd32e None None] Exception during message handling
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 282, in _process_data
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp ctxt.reply(rval, None, connection_pool=self.connection_pool)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 191, in reply
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp ending, log_failure)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 171, in msg_reply
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp conn.direct_send(msg_id, rpc_common.serialize_msg(msg))
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 692, in direct_send
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.publisher_send(DirectPublisher, msg_id, msg)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 668, in publisher_send
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.ensure(_error_callback, _publish)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 554, in ensure
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp return method(*args, **kwargs)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 665, in _publish
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp publisher = cls(self.conf, self.channel, topic, **kwargs)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 323, in __init__
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp type='direct', **options)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 295, in __init__
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.reconnect(channel)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 303, in reconnect
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp routing_key=self.routing_key)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 83, in __init__
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.revive(self._channel)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 213, in revive
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.declare()
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/messaging.py", line 100, in declare
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp self.exchange.declare()
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/kombu/entity.py", line 159, in declare
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp passive=passive)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 600, in exchange_declare
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp (40, 11), # Channel.exchange_declare_ok
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 70, in wait
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp return self.dispatch_method(method_sig, args, content)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp return amqp_method(self, args)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp File "/usr/local/lib/python2.7/dist-packages/amqp/channel.py", line 221, in _close
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp raise ChannelError(reply_code, reply_text, (class_id, method_id))
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp ChannelError: 406: (PRECONDITION_FAILED - cannot redeclare exchange '49e38540308447d1b64841b9e8a31e5a' in vhost '/' with different type, durable, internal or autodelete value, (40, 10), None)
2013-02-07 15:48:34.709 7496 TRACE nova.openstack.common.rpc.amqp
2013-02-07 15:48:34.714 ERROR nova.openstack.common.rpc.common [req-d7094194-b6c4-4784-b265-a401dbffd32e None None] Returning exception 406: (PRECONDITION_FAILED - cannot redeclare exchange '49e38540308447d1b64841b9e8a31e5a' in vhost '/' with different type, durable, internal or autodelete value, (40, 10), None) to caller

This was working fine until a day or two ago.

[Jacob Godin (jacobgodin)]

Here's what an example exchange looks like in Rabbit:

# rabbitmqctl list_exchanges | grep df2d38da2f454724920c2810bd0cf663
df2d38da2f454724920c2810bd0cf663 direct

[Jacob Godin (jacobgodin)]

rabbitmq.log

connection <0.14457.55>, channel 1 - error:
{amqp_error,precondition_failed,
            "cannot redeclare exchange 'bbd487ff5d6b4ab3903e348254c577b6' in vhost '/' with different type, durable, internal or autodelete value",
            'exchange.declare'}

[Vish Ishaya (vishvananda)]

Are you still seeing this? I suspect a packaging bug here. Perhaps an old version of consoleauth is getting run or it has incorrect settings?

[Jacob Godin (jacobgodin)]

Nope, seems to be working fine now. Have a new install since this was submitted.