OpenStack Compute (nova)

novnc console failure after Icehouse upgrade

Bug #1317174 reported by Adam Huffman on 2014-05-07

This bug affects 10 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	Undecided	Unassigned	OpenStack Compute (nova) 2014.2 "juno"

Bug Description

After upgrading my Havana installation to Icehouse, VNC console logins are no longer working (1006 error).

The version in use is:

openstack-nova-novncproxy-2014.1-2.el6.noarch

from RDO.

This is the full error in the logs:

2014-05-07 17:12:58.003 13074 AUDIT nova.consoleauth.manager [req-684f9e8d-3c0a-4647-aa66-44f0bb35c4df None None] Checking Token: dbbd1b9b-002f-46b6-bf79-0d90e92c034e, True
2014-05-07 17:12:58.112 13074 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: tuple index out of range
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
incoming.message))

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function
args = (_load_instance(args[0]),) + args[1:]

IndexError: tuple index out of range
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher if self._validate_token(context, token):
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher token['console_type'])
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher console_type=console_type)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher wait_for_reply=True, timeout=timeout)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher timeout=timeout)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher raise result
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher IndexError: tuple index out of range
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher args = (_load_instance(args[0]),) + args[1:]
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher IndexError: tuple index out of range
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher
2014-05-07 17:12:58.112 13074 TRACE oslo.messaging.rpc.dispatcher

Tags:

Revision history for this message

Samuel Chen (samuel.chen) wrote on 2014-05-22:

Download full text (6.9 KiB)

Same issue in my env when I upgraded my Havana + CentOS 6.5 to Icehouse
This is my logs:

2014-05-23 00:36:20.953 838 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: tuple index out of range
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
incoming.message))

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function
args = (_load_instance(args[0]),) + args[1:]

IndexError: tuple index out of range
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher if self._validate_token(context, token):
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher token['console_type'])
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher console_type=console_type)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher wait_for_reply=True, timeout=timeout)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.d...

Same issue in my env when I upgraded my Havana + CentOS 6.5 to Icehouse
This is my logs:

2014-05-23 00:36:20.953 838 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: tuple index out of range
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)

File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function
    args = (_load_instance(args[0]),) + args[1:]

IndexError: tuple index out of range
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     incoming.message))
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     if self._validate_token(context, token):
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     token['console_type'])
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     console_type=console_type)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     wait_for_reply=True, timeout=timeout)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     timeout=timeout)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     return self._send(target, ctxt, message, wait_for_reply, timeout)
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher   File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher     raise result
2014-05-23 00:36:20.953 838 TRACE oslo.messaging.rpc.dispatcher IndexError: tuple index out of range
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)

File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function
    args = (_load_instance(args[0]),) + args[1:]

IndexError: tuple index out of range
 to caller
2014-05-23 00:36:20.955 838 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token\n    if self._validate_token(context, token):\n', '  File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token\n    token[\'console_type\'])\n', '  File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port\n    console_type=console_type)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n    wait_for_reply=True, timeout=timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send\n    timeout=timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send\n    raise result\n', 'IndexError: tuple index out of range\nTraceback (most recent call last):\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n\n  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 390, in decorated_function\n    args = (_load_instance(args[0]),) + args[1:]\n\nIndexError: tuple index out of range\n\n']

Revision history for this message

Andres Toomsalu (andres-active) wrote on 2014-08-24:

Download full text (3.4 KiB)

Same situation - after upgrade from RDO Havana to Icehouse (CentOS 6.5) VNC console is failing with error 1006.

from the /var/log/nova/consoleauth.log:

2014-08-24 23:15:46.140 22506 ERROR oslo.messaging._drivers.common [-] Returning exception tuple index out of range
Traceback (most recent call last):

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
incoming.message))

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
return self._do_dispatch(endpoint, method, ctxt, args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
result = getattr(endpoint, method)(ctxt, **new_args)

File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
return func(*args, **kwargs)

File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 384, in decorated_function
args = (_load_instance(args[0]),) + args[1:]

IndexError: tuple index out of range
to caller
2014-08-24 23:15:46.140 22506 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token\n if self._validate_token(context, token):\n', ' File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token\n token[\'console_type\'])\n', ' File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port\n console_type=console_type)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n wait_for_reply=True, timeout=timeout)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send\n timeout=timeout)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n return self._send(target, ctxt, message, wait_for_reply, timeout)\n', ' File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send\n raise result\n', 'IndexError: tuple index out of range\nTraceback (most recent call last):\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n incoming.message))\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n\n File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n\n File "/usr/lib/python2....

Same situation - after upgrade from RDO Havana to Icehouse (CentOS 6.5) VNC console is failing with error 1006.

from the /var/log/nova/consoleauth.log:

2014-08-24 23:15:46.140 22506 ERROR oslo.messaging._drivers.common [-] Returning exception tuple index out of range
Traceback (most recent call last):
 
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
    incoming.message))
 
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
    return self._do_dispatch(endpoint, method, ctxt, args)
 
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
    result = getattr(endpoint, method)(ctxt, **new_args)
 
  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
    return func(*args, **kwargs)
 
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 384, in decorated_function
    args = (_load_instance(args[0]),) + args[1:]
 
IndexError: tuple index out of range
 to caller
2014-08-24 23:15:46.140 22506 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n', '  File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 117, in check_token\n    if self._validate_token(context, token):\n', '  File "/usr/lib/python2.6/site-packages/nova/consoleauth/manager.py", line 108, in _validate_token\n    token[\'console_type\'])\n', '  File "/usr/lib/python2.6/site-packages/nova/compute/rpcapi.py", line 506, in validate_console_port\n    console_type=console_type)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/client.py", line 150, in call\n    wait_for_reply=True, timeout=timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/transport.py", line 90, in _send\n    timeout=timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 412, in send\n    return self._send(target, ctxt, message, wait_for_reply, timeout)\n', '  File "/usr/lib/python2.6/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 405, in _send\n    raise result\n', 'IndexError: tuple index out of range\nTraceback (most recent call last):\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply\n    incoming.message))\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n    return self._do_dispatch(endpoint, method, ctxt, args)\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n    result = getattr(endpoint, method)(ctxt, **new_args)\n\n  File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner\n    return func(*args, **kwargs)\n\n  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 384, in decorated_function\n    args = (_load_instance(args[0]),) + args[1:]\n\nIndexError: tuple index out of range\n\n']

Revision history for this message

Andres Toomsalu (andres-active) wrote on 2014-08-24:

Package version was : openstack-nova-novncproxy.noarch 2014.1.1-3.el6

Revision history for this message

Andres Toomsalu (andres-active) wrote on 2014-08-25:

This workaround worked for me:

# on Controller host:
# remove icehouse-compat from nova.conf
openstack-config --del /etc/nova/nova.conf upgrade_levels compute

# restart nova services on Controller host
SVC_NOVA="openstack-nova-api openstack-nova-cert openstack-nova-scheduler openstack-nova-conductor openstack-nova-novncproxy openstack-nova-consoleauth"
for SVC in $SVC_NOVA; do service $SVC restart; done

Thats a workaround - yet not proper solution - as if upgrading Compute hosts one-by-one icehouse-compat has to be preserved until ALL hosts have been upgraded to icehouse - if Im getting this correct. So novnc consoles would be still out during the upgrade process for Computes running already on icehouse.

Revision history for this message

Sam Morrison (sorrison) wrote on 2014-08-25:

This is fixed for me, the latest code in stable/icehouse branch works, not sure the exact commit that fixed it though sorry.

Revision history for this message

Sean Dague (sdague) wrote on 2014-09-12:

It is reporting as fixed

Changed in nova:
status:	New → Fix Committed

Thierry Carrez (ttx) on 2014-10-01

Changed in nova:
milestone:	none → juno-rc1
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-10-16

Changed in nova:
milestone:	juno-rc1 → 2014.2

Revision history for this message

Prashant (pczanwar) wrote on 2015-08-11:

I have hit this issue just recently, and solution what's recommended, only service what needs to be restarted on controller nodes is "consoleauth"

This workaround worked for me:

# on Controller host:
# remove icehouse-compat from nova.conf
openstack-config --del /etc/nova/nova.conf upgrade_levels compute

# restart nova services on Controller host
openstack-service restart openstack-nova-consoleauth

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.