nova-compute no longer work after reboot with instances running

[Yun Mao]

How to reproduce:

install from the master branch. Make sure resume_guests_state_on_host_boot=True

Start an instance. Once the instance is in running, reboot the compute node.

After reboot, the nova-compute daemon can no longer start. Here is the traceback:

2013-02-18 20:13:10.703 INFO nova.compute.manager [req-2399728a-770e-469a-98d2-018caa9cc482 None None] [instance:
 3422520b-0c31-4f99-8cb0-8c6416a5f15a] Rebooting instance after nova-compute restart.
2013-02-18 20:13:10.703 DEBUG nova.openstack.common.rpc.amqp [req-2399728a-770e-469a-98d2-018caa9cc482 None None]
 Making synchronous call on conductor ... multicall /usr/local/lib/python2.7/dist-packages/nova/openstack/common/
rpc/amqp.py:362
2013-02-18 20:13:10.703 DEBUG nova.openstack.common.rpc.amqp [req-2399728a-770e-469a-98d2-018caa9cc482 None None]
 MSG_ID is 387dff66e09f4c489a53afbafb9d8906 multicall /usr/local/lib/python2.7/dist-packages/nova/openstack/commo
n/rpc/amqp.py:365
2013-02-18 20:13:10.779 1291 CRITICAL nova [-] string indices must be integers
Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)

  File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 133, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py", line 235, in block_device_mapping_get_
all_by_instance
    context, instance['uuid'])

TypeError: string indices must be integers
2013-02-18 20:13:10.779 1291 TRACE nova Traceback (most recent call last):
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/bin/nova-compute", line 84, in <module>
2013-02-18 20:13:10.779 1291 TRACE nova service.wait()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 209
, in wait
2013-02-18 20:13:10.779 1291 TRACE nova super(ServiceLauncher, self).wait()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 179
, in wait
2013-02-18 20:13:10.779 1291 TRACE nova service.wait()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 168, in wait
2013-02-18 20:13:10.779 1291 TRACE nova return self._exit_event.wait()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/event.py", line 1
16, in wait
2013-02-18 20:13:10.779 1291 TRACE nova return hubs.get_hub().switch()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py", lin
e 187, in switch
2013-02-18 20:13:10.779 1291 TRACE nova return self.greenlet.switch()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 194, in main
2013-02-18 20:13:10.779 1291 TRACE nova result = function(*args, **kwargs)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 147
, in run_server
2013-02-18 20:13:10.779 1291 TRACE nova server.start()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/service.py", line 429
, in start
2013-02-18 20:13:10.779 1291 TRACE nova self.manager.init_host()
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
line 516, in init_host
2013-02-18 20:13:10.779 1291 TRACE nova self._init_instance(context, instance)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
line 475, in _init_instance
2013-02-18 20:13:10.779 1291 TRACE nova context, instance['uuid'])
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
line 1039, in _get_instance_volume_block_device_info
2013-02-18 20:13:10.779 1291 TRACE nova bdms = self._get_instance_volume_bdms(context, instance)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/compute/manager.py",
line 1023, in _get_instance_volume_bdms
2013-02-18 20:13:10.779 1291 TRACE nova context, instance))
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/conductor/api.py", li
ne 535, in block_device_mapping_get_all_by_instance
2013-02-18 20:13:10.779 1291 TRACE nova context, instance)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/conductor/rpcapi.py",
 line 233, in block_device_mapping_get_all_by_instance
2013-02-18 20:13:10.779 1291 TRACE nova return self.call(context, msg, version='1.13')
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
proxy.py", line 80, in call
2013-02-18 20:13:10.779 1291 TRACE nova return rpc.call(context, self._get_topic(topic), msg, timeout)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
__init__.py", line 139, in call
2013-02-18 20:13:10.779 1291 TRACE nova return _get_impl().call(CONF, context, topic, msg, timeout)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
impl_kombu.py", line 772, in call
2013-02-18 20:13:10.779 1291 TRACE nova rpc_amqp.get_connection_pool(conf, Connection))
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
amqp.py", line 379, in call
2013-02-18 20:13:10.779 1291 TRACE nova rv = list(rv)
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
amqp.py", line 347, in __iter__
2013-02-18 20:13:10.779 1291 TRACE nova raise result
2013-02-18 20:13:10.779 1291 TRACE nova TypeError: string indices must be integers
2013-02-18 20:13:10.779 1291 TRACE nova Traceback (most recent call last):
2013-02-18 20:13:10.779 1291 TRACE nova
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
amqp.py", line 276, in _process_data
2013-02-18 20:13:10.779 1291 TRACE nova rval = self.proxy.dispatch(ctxt, version, method, **args)
2013-02-18 20:13:10.779 1291 TRACE nova
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/openstack/common/rpc/
dispatcher.py", line 133, in dispatch
2013-02-18 20:13:10.779 1291 TRACE nova return getattr(proxyobj, method)(ctxt, **kwargs)
2013-02-18 20:13:10.779 1291 TRACE nova
2013-02-18 20:13:10.779 1291 TRACE nova File "/usr/local/lib/python2.7/dist-packages/nova/conductor/manager.py"
, line 235, in block_device_mapping_get_all_by_instance
2013-02-18 20:13:10.779 1291 TRACE nova context, instance['uuid'])
2013-02-18 20:13:10.779 1291 TRACE nova
2013-02-18 20:13:10.779 1291 TRACE nova TypeError: string indices must be integers
2013-02-18 20:13:10.779 1291 TRACE nova

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/22235

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/22235
Committed: http://github.com/openstack/nova/commit/376dafd32f8b985e2af556315d9f33cfb62df96f
Submitter: Jenkins
Branch: master

commit 376dafd32f8b985e2af556315d9f33cfb62df96f
Author: Yun Mao <email address hidden>
Date: Mon Feb 18 15:25:01 2013 -0500

    Fix _get_instance_volume_block_device_info call parameter

    _get_instance_volume_block_device_info() is expecting an
    instance dict-like object instead of instance_uuid

    Fix bug 1129441

    Change-Id: Id4d536553ce9858434c377653fa79562e1282a34