Ironic: "nova delete" doesn't work when instance is spawning

Bug #1455000 reported by Lucas Alvares Gomes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
Lucas Alvares Gomes

Bug Description

The "nova delete" command can be issued while the instance is spawning and even Ironic support aborting a deployment mid-operation (when node is in WAITDEPLOY state).

But if you issue a "nova delete" while the instance is spawming with the Ironic driver the instance will be stuck there and nothing will happen. That's because of a lock[1], basically the Ironic driver won't return from spawn() until the node have been successfully deployed or error out and that's holding the lock.

WORKAROUND:

As a work around, you can issue "nova delete" to the instance and that will mark the task_state as "deleting". And then you can delete the instance directly via the Ironic API as well by doing:

$ ironic node-set-provision-state <node uuid> deleted

That will realease the instance.

[1] https://github.com/openstack/nova/blob/3841b8254e0dd820d26377312b6c004f4bd9b58e/nova/compute/manager.py#L2431

Tags: ironic
Changed in nova:
assignee: nobody → Lucas Alvares Gomes (lucasagomes)
tags: added: ironic
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/182992

Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/182992
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ea3967a1fb47297608defd680286fd9051ff5bbe
Submitter: Jenkins
Branch: master

commit ea3967a1fb47297608defd680286fd9051ff5bbe
Author: Lucas Alvares Gomes <email address hidden>
Date: Thu May 14 10:56:13 2015 +0100

    Ironic: Fix delete instance when spawning

    Nova allows issuing a "nova delete" command when the instance is being
    spawned, that wasn't working in Ironic because the spawn() in the Ironic
    nova driver doesn't return until the node have been deployed or errored
    out and that causes Ironic to hold a lock that prevents the instance
    termination to call destroy() in the Ironic nova driver.

    This patch fixes that problem by making the loop waiting on the node to
    get active (or error) in Ironic to also look at the instance task_state
    and see if it's being deleted. In case the instance is being deleted
    mid deployment operation we break the loop releasing the lock and then
    destroy() gets called in Ironic aborting the deployment there.

    Closes-Bug: #1455000
    Change-Id: I315b8430a92907d27534f7b4828c35792371d6dc

Changed in nova:
status: In Progress → Fix Committed
aeva black (tenbrae)
Changed in nova:
importance: Undecided → Medium
summary: - Ironic: "nova delete" doesn't work when instance is spawming
+ Ironic: "nova delete" doesn't work when instance is spawning
Thierry Carrez (ttx)
Changed in nova:
milestone: none → liberty-1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: liberty-1 → 12.0.0
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.