nova rebuild fails after two rebuild requests when ironic is used

Bug #1564921 reported by Vladyslav Drok on 2016-04-01
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ironic
Undecided
Vladyslav Drok
OpenStack Compute (nova)
Undecided
Vladyslav Drok
Mitaka
Undecided
Unassigned
Newton
Undecided
Unassigned

Bug Description

First nova rebuild request passes fine, but further requests fail with the following message:

Instance b460e640-e601-4e68-b0e8-231e15201412 is already associated with a node, it cannot be associated with this other node 10c0b922-cb39-412e-849a-27e66042d4c0 (HTTP 409)", "code": 500, "details": " File \"/opt/stack/nova/nova/compute/manager.py\"

The reason for this is that nova tries to reshcedule an instance during rebuild, and in case of ironic, there can't be 2 nodes associated with the same instance_uuid.

This can be checked on devstack since change I0233f964d8f294f0ffd9edcb16b1aaf93486177f that introduced it with ironic virt driver and neutron.

Vladyslav Drok (vdrok) on 2016-04-01
Changed in nova:
assignee: nobody → Vladyslav Drok (vdrok)
Matt Riedemann (mriedem) wrote :

I'm not really sure what I0233f964d8f294f0ffd9edcb16b1aaf93486177f has to do with this.

tags: added: ironic rebuild
Changed in nova:
status: New → Confirmed
Vladyslav Drok (vdrok) on 2016-04-14
Changed in ironic:
assignee: nobody → Vladyslav Drok (vdrok)

Fix proposed to branch: master
Review: https://review.openstack.org/306010

Changed in nova:
status: Confirmed → In Progress
Changed in ironic:
status: New → In Progress

Change abandoned by Vladyslav Drok (<email address hidden>) on branch: master
Review: https://review.openstack.org/306011

Vladyslav Drok (vdrok) on 2016-04-15
Changed in ironic:
status: In Progress → Won't Fix
Matt Riedemann (mriedem) on 2016-09-23
tags: added: mitaka-backport-potential newton-backport-potential

Reviewed: https://review.openstack.org/306010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54b122caec1ae2418fc3e296be604d072cb5815a
Submitter: Jenkins
Branch: master

commit 54b122caec1ae2418fc3e296be604d072cb5815a
Author: Vladyslav Drok <email address hidden>
Date: Thu Apr 14 21:15:10 2016 +0300

    Update instance node on rebuild only when it is recreate

    When using ironic virt driver, if scheduled_node is not specified
    in rebuild_instance compute manager method (as it happens in case
    of instance rebuild), the first ironic node is selected:

     computes = ComputeNodeList.get_all_by_host(context, host, use_slave)
     return computes[0]

    After the first rebuild, instance.node is updated to be this first
    ironic node, which causes subsequent rebuilds to fail, as virt driver
    tries to set instance_uuid on a newly selected ironic node and fails.

    Closes-bug: #1564921
    Change-Id: I2fe6e439135ba6aa4120735d030ced31081ef202

Changed in nova:
status: In Progress → Fix Released

Reviewed: https://review.openstack.org/375715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97d784b454baf3289ce2f3e1812fb94c8469b136
Submitter: Jenkins
Branch: stable/newton

commit 97d784b454baf3289ce2f3e1812fb94c8469b136
Author: Vladyslav Drok <email address hidden>
Date: Thu Apr 14 21:15:10 2016 +0300

    Update instance node on rebuild only when it is recreate

    When using ironic virt driver, if scheduled_node is not specified
    in rebuild_instance compute manager method (as it happens in case
    of instance rebuild), the first ironic node is selected:

     computes = ComputeNodeList.get_all_by_host(context, host, use_slave)
     return computes[0]

    After the first rebuild, instance.node is updated to be this first
    ironic node, which causes subsequent rebuilds to fail, as virt driver
    tries to set instance_uuid on a newly selected ironic node and fails.

    Closes-bug: #1564921
    Change-Id: I2fe6e439135ba6aa4120735d030ced31081ef202
    (cherry picked from commit 54b122caec1ae2418fc3e296be604d072cb5815a)

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

This issue was fixed in the openstack/nova 14.0.3 release.

To post a comment you must log in.
This report contains Public information  Edit
Everyone can see this information.

Duplicates of this bug

Other bug subscribers