inconsistent nova-bm state will prevent launching new instances

Bug #1096722 reported by Robert Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Critical
Robert Collins

Bug Description

if the nova-bm state lists a deleted instance against a baremetal node, then
nova/virt/driver.py", line 160, in instance_exists
will throw:
InstanceNotFound_Remote: Instance 2c9b1224-bbbf-47a8-bbce-c9be61ac554b could not be found.

where the instance id will be that of the deleted instance.

This is a horribly inefficient implementation anyhow, its iterating all baremetal nodes to check for the desired one not existing.

This suggests we should implement instance_exists more directly, *and* we should make
baremetal list_instances have a try: except around the call to instance_get_by_uuid.

Tags: baremetal
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/19069

Changed in nova:
assignee: nobody → Robert Collins (lifeless)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/19069
Committed: http://github.com/openstack/nova/commit/37293192c6ae7b841ded4fbccd4d5b36cc89fbaa
Submitter: Jenkins
Branch: master

commit 37293192c6ae7b841ded4fbccd4d5b36cc89fbaa
Author: Robert Collins <email address hidden>
Date: Mon Jan 7 15:17:42 2013 +1300

    Cope better with out of sync bm data.

    The baremetal hypervisor tracks references to instances which are
    deleted asynchronously from the commit to the nova bm. As such, its a
    normal but rare condition for a bare metal node to refer to a deleted
    (and even garbage collected) nova instance. We should treat such
    instances are deleted rather than erroring on any call through
    list_instances(), permitting starting new instances and listing
    instances to work - fixing bug: 1096722.

    Pathologically, the database may suffer permanent skew, which means we
    need a way to fix it (and thats tracked as a separate bug).

    Change-Id: Ic21ff66b1fc0ad64bb5feff26291873b96d20e4e

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-3
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.