Compute _init_instance changes instance from object to dict causing an AttributeError

[Andrew Laski]

On a compute restart sometimes the following error occurs and the compute fails to start:

2014-04-17 03:36:09.527 26115 ERROR nova.openstack.common.threadgroup [-] 'dict' object has no attribute 'task_state'
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup x.wait()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/openstack/common/service.py", line 480, in run_service
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup service.start()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/service.py", line 177, in start
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup self.manager.init_host()
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 871, in init_host
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup self._init_instance(context, instance)
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup File "/opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/compute/manager.py", line 731, in _init_instance
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup if instance.task_state == task_states.DELETING:
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup AttributeError: 'dict' object has no attribute 'task_state'
2014-04-17 03:36:09.527 26115 TRACE nova.openstack.common.threadgroup
2014-04-17 04:01:15.145 27377 DEBUG nova.servicegroup.api [-] ServiceGroup driver defined as an instance of db __new__ /opt/rackstack/615.9/nova/lib/python2.6/site-packages/nova/servicegroup/api.py:62

The issue appears to lie with the line immediately above which performs an instance update and overwrites the instance object with a dict.
instance = self._instance_update(context, instance.uuid, task_state=None)

[Andrew Laski (alaski)]

After the instance update the next compute start will succeed because it doesn't enter the same if block.

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: master
Review: https://review.openstack.org/88400

[Matt Riedemann (mriedem)]

This was a regression in Icehouse: https://review.openstack.org/#/c/57967/

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88400
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b4b5254a56d808f2582784897364de1adafeb43a
Submitter: Jenkins
Branch: master

commit b4b5254a56d808f2582784897364de1adafeb43a
Author: Andrew Laski <email address hidden>
Date: Thu Apr 17 14:51:06 2014 -0400

    Don't overwrite instance object with dict in _init_instance()

    If a compute service is started while it has an instance in task_state
    REBOOT_STARTED or REBOOT_STARTED_HARD it will trigger an instance update
    which will overwrite the instance object with an instance dict. This
    causes an AttributeError on the next line when instance.task_state is
    referenced. This seems like a simple oversight and should be switched
    to using the instance.save method.

    Change-Id: I44c3eab2c8821b66b0b9ef4696e56bd577ae8fed
    Closes-Bug: 1309067

[Openstack Gerrit (openstack-gerrit)]

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/88697

[Openstack Gerrit (openstack-gerrit)]

Reviewed: https://review.openstack.org/88697
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=7f9f3efc45a2b458e37ba391dbad763a02fca0f6
Submitter: Jenkins
Branch: stable/icehouse

commit 7f9f3efc45a2b458e37ba391dbad763a02fca0f6
Author: Andrew Laski <email address hidden>
Date: Thu Apr 17 14:51:06 2014 -0400

    Don't overwrite instance object with dict in _init_instance()

    If a compute service is started while it has an instance in task_state
    REBOOT_STARTED or REBOOT_STARTED_HARD it will trigger an instance update
    which will overwrite the instance object with an instance dict. This
    causes an AttributeError on the next line when instance.task_state is
    referenced. This seems like a simple oversight and should be switched
    to using the instance.save method.

    Change-Id: I44c3eab2c8821b66b0b9ef4696e56bd577ae8fed
    Closes-Bug: 1309067
    (cherry picked from commit b4b5254a56d808f2582784897364de1adafeb43a)