nova rebuild fails after two rebuild requests when ironic is used

Bug #1564921 reported by Vladyslav Drok
22
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Ironic
Won't Fix
Undecided
Vladyslav Drok
OpenStack Compute (nova)
Fix Released
Undecided
Vladyslav Drok
Mitaka
New
Undecided
Unassigned
Newton
Fix Committed
Undecided
Unassigned

Bug Description

First nova rebuild request passes fine, but further requests fail with the following message:

Instance b460e640-e601-4e68-b0e8-231e15201412 is already associated with a node, it cannot be associated with this other node 10c0b922-cb39-412e-849a-27e66042d4c0 (HTTP 409)", "code": 500, "details": " File \"/opt/stack/nova/nova/compute/manager.py\"

The reason for this is that nova tries to reshcedule an instance during rebuild, and in case of ironic, there can't be 2 nodes associated with the same instance_uuid.

This can be checked on devstack since change I0233f964d8f294f0ffd9edcb16b1aaf93486177f that introduced it with ironic virt driver and neutron.

Vladyslav Drok (vdrok)
Changed in nova:
assignee: nobody → Vladyslav Drok (vdrok)
Revision history for this message
Matt Riedemann (mriedem) wrote :

I'm not really sure what I0233f964d8f294f0ffd9edcb16b1aaf93486177f has to do with this.

tags: added: ironic rebuild
Changed in nova:
status: New → Confirmed
Vladyslav Drok (vdrok)
Changed in ironic:
assignee: nobody → Vladyslav Drok (vdrok)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/306010

Changed in nova:
status: Confirmed → In Progress
Changed in ironic:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (master)

Fix proposed to branch: master
Review: https://review.openstack.org/306011

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on ironic (master)

Change abandoned by Vladyslav Drok (<email address hidden>) on branch: master
Review: https://review.openstack.org/306011

Vladyslav Drok (vdrok)
Changed in ironic:
status: In Progress → Won't Fix
Matt Riedemann (mriedem)
tags: added: mitaka-backport-potential newton-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/306010
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=54b122caec1ae2418fc3e296be604d072cb5815a
Submitter: Jenkins
Branch: master

commit 54b122caec1ae2418fc3e296be604d072cb5815a
Author: Vladyslav Drok <email address hidden>
Date: Thu Apr 14 21:15:10 2016 +0300

    Update instance node on rebuild only when it is recreate

    When using ironic virt driver, if scheduled_node is not specified
    in rebuild_instance compute manager method (as it happens in case
    of instance rebuild), the first ironic node is selected:

     computes = ComputeNodeList.get_all_by_host(context, host, use_slave)
     return computes[0]

    After the first rebuild, instance.node is updated to be this first
    ironic node, which causes subsequent rebuilds to fail, as virt driver
    tries to set instance_uuid on a newly selected ironic node and fails.

    Closes-bug: #1564921
    Change-Id: I2fe6e439135ba6aa4120735d030ced31081ef202

Changed in nova:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/newton)

Reviewed: https://review.openstack.org/375715
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=97d784b454baf3289ce2f3e1812fb94c8469b136
Submitter: Jenkins
Branch: stable/newton

commit 97d784b454baf3289ce2f3e1812fb94c8469b136
Author: Vladyslav Drok <email address hidden>
Date: Thu Apr 14 21:15:10 2016 +0300

    Update instance node on rebuild only when it is recreate

    When using ironic virt driver, if scheduled_node is not specified
    in rebuild_instance compute manager method (as it happens in case
    of instance rebuild), the first ironic node is selected:

     computes = ComputeNodeList.get_all_by_host(context, host, use_slave)
     return computes[0]

    After the first rebuild, instance.node is updated to be this first
    ironic node, which causes subsequent rebuilds to fail, as virt driver
    tries to set instance_uuid on a newly selected ironic node and fails.

    Closes-bug: #1564921
    Change-Id: I2fe6e439135ba6aa4120735d030ced31081ef202
    (cherry picked from commit 54b122caec1ae2418fc3e296be604d072cb5815a)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 15.0.0.0b1

This issue was fixed in the openstack/nova 15.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 14.0.3

This issue was fixed in the openstack/nova 14.0.3 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.