vmware: nova-compute will not start if some instance relying datastore not available

Bug #1337236 reported by zhu zhu
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
zhu zhu
Fix Committed

Bug Description

First use vcenter driver to spawn some instances to one of the datastores that esxi is binding to. Later this datastore became unavailable due to certain reason(power off or network problem). Then when restart nova-compute, found that compute service will exit with errors. This will openstack compute not usable.

2014-07-03 01:38:13.961 3634 DEBUG nova.compute.manager [req-11bc0618-8696-464d-8820-7565db8f44c3 None None] [instance: 9428cf95-5
37f-48f6-b79e-faa981f6066d] NV-AC7AA80 Checking state _get_power_state /usr/lib/python2.6/site-packages/nova/compute/manager.py:10
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/eventlet/hubs/poll.py", line 97, in wait
    readers.get(fileno, noop).cb(fileno)
  File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 480, in run_service
  File "/usr/lib/python2.6/site-packages/nova/service.py", line 180, in start
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1037, in init_host
    self._init_instance(context, instance)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 865, in _init_instance
    try_reboot, reboot_type = self._retry_reboot(context, instance)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 963, in _retry_reboot
    current_power_state = self._get_power_state(context, instance)
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1056, in _get_power_state
    return self.driver.get_info(instance)["state"]
  File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 862, in get_info
    return _vmops.get_info(instance)
  File "/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1376, in get_info
    max_mem = int(query['summary.config.memorySizeMB']) * 1024
KeyError: 'summary.config.memorySizeMB'

Revision history for this message
zhu zhu (zhuzhubj) wrote :
Changed in nova:
assignee: nobody → zhu zhu (zhuzhubj)
Changed in nova:
status: New → In Progress
Gary Kotton (garyk)
Changed in nova:
importance: Undecided → High
tags: added: icehouse-backport-potential vmware
Revision history for this message
Qin Zhao (zhaoqin) wrote :

It seems to be a must fix one. Thanks to Zhu Zhu for reporting it!

description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/104495
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=488f88c4d021bf429f62ca46dd8299a70d31505e
Submitter: Jenkins
Branch: master

commit 488f88c4d021bf429f62ca46dd8299a70d31505e
Author: ZHU ZHU <email address hidden>
Date: Thu Jul 3 03:19:19 2014 -0500

    VMWare: Fix nova-compute crash when instance datastore not available

    During nova-compute service startup, init_host will try to get_info for
    instances. If datastore of certain instances become not reachable, some
    VM properties like 'summary.config.memorySizeMB' will not be available.
    This will cause nova-compute process exit with error.

    Change-Id: I6753ce5658b629c216ebfbaab610c8fd0755466a
    Closes-Bug: #1337236

Changed in nova:
status: In Progress → Fix Committed
Tracy Jones (tjones-i)
Changed in openstack-vmwareapi-team:
status: New → Fix Committed
importance: Undecided → High
Thierry Carrez (ttx)
Changed in nova:
milestone: none → kilo-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: kilo-1 → 2015.1.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.