Counting vcpus can fail when domains are not complete

Bug #1153694 reported by Stanislaw Pitucha
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Stanislaw Pitucha

Bug Description

Libvirt allows the virDomainGetVcpus() function to fail in case the instance is not ready. This can cause random failures during instance startup or shutdown.

Traceback returned in that case is:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 779, in _run_instance
    requested_networks, macs, security_groups)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1002, in _allocate_network
    expected_task_state=None)
  File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 374, in _instance_update
    instance_ref['node'] in self.driver.get_available_nodes()):
  File "/usr/lib/python2.7/dist-packages/nova/virt/driver.py", line 843, in get_available_nodes
    stats = self.get_host_stats(refresh=True)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3367, in get_host_stats
    return self.host_state.get_host_stats(refresh=refresh)
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3662, in get_host_stats
    self.update_status()
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3670, in update_status
    data["vcpus_used"] = self.driver.get_vcpu_used()
  File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2556, in get_vcpu_used
    total += len(vcpus[1])
TypeError: 'NoneType' object has no attribute '__getitem__'

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/24094

Changed in nova:
assignee: nobody → Stanislaw Pitucha (stanislaw-pitucha)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/24094
Committed: http://github.com/openstack/nova/commit/f97f6214dac7951cb4d91bf24e9ceb3a43e65938
Submitter: Jenkins
Branch: master

commit f97f6214dac7951cb4d91bf24e9ceb3a43e65938
Author: Stanislaw Pitucha <email address hidden>
Date: Mon Mar 11 17:54:47 2013 +0000

    Handle vcpu counting failures gracefully

    The virDomainGetVcpus() call in libvirt can fail in some cases. This
    should be handled properly in nova and not cause failures in starting
    up new instances.

    Fixes bug 1153694

    Change-Id: I8e072c53ee2bafcb50083d793ceabe5292038ab6

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-rc1 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.