Ironic: Deleting while spawning can leave orphan ACTIVE nodes in Ironic
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Won't Fix
|
Medium
|
Lucas Alvares Gomes | ||
OpenStack Compute (nova) |
Fix Released
|
Medium
|
Unassigned |
Bug Description
The Ironic nova driver won't try to delete the instance in Ironic if the node's provision state is DEPLOYING [1] , this is known to fail with the current Ironic code because we just can't abort the installation at the DEPLOYING stage.
But the Ironic nova driver just keep going and tries to clean up the deployment environment (without telling Ironic to unprovision the instance) and it will fail as well. But the the code that cleans up the instance will keep retrying [3] because there's a transition in progress and it can't update the node. But when the node finishes the deployment, if the retrying didn't timed out, the destroy() method from the Nova driver will succeed cleaning deployment environment and the Nova instance will be deleted but the Ironic node will continue to marked as ACTIVE in Ironic and now orphan because there's no instance in Nova associated with it [4]
The good news is that since nova clean up the network stuff the instance won't be accessible.
WORKAROUND:
Unprovision the node using the Ironic API directly
$ ironic node-set-
PROPOSED FIX:
IMO the ironic nova driver should try to tell Ironic to delete the instance even when the provision state of the node is DEPLOYING. If it fails the nova delete command will fail saying it can not delete the instance, which is fine until this gets resolved in Ironic (there's work going on to be able to abort a deployment at any stage)
[3] From the nova-compute logs
{"error_message": "{\"debuginfo\": null, \"faultcode\": \"Client\", \"faultstring\": \"Node d240ae0d-
from (pid=6672) log_http_response /usr/local/
2015-07-23 11:07:40.358 WARNING ironicclient.
2015-07-23 11:07:40.358 WARNING ironicclient.
Changed in nova: | |
assignee: | nobody → Lucas Alvares Gomes (lucasagomes) |
summary: |
- Ironic: Deleting while spawnming can leave orphan ACTIVE nodes in Ironic + Ironic: Deleting while spawning can leave orphan ACTIVE nodes in Ironic |
Changed in nova: | |
assignee: | Lucas Alvares Gomes (lucasagomes) → John Garbutt (johngarbutt) |
Changed in nova: | |
assignee: | John Garbutt (johngarbutt) → Lucas Alvares Gomes (lucasagomes) |
Changed in nova: | |
importance: | Undecided → Medium |
Changed in nova: | |
assignee: | Lucas Alvares Gomes (lucasagomes) → nobody |
Changed in nova: | |
status: | In Progress → Confirmed |
@Lucas Alvares Gomes (lucasagomes):
Since you are set as assignee, I switch the status to "In Progress".