Node stuck on DEPLOYING (potentially all *ING) state(s)
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Ironic |
Fix Released
|
High
|
Lucas Alvares Gomes |
Bug Description
If a conductor deploying a node dies due some OOM Killer (or whatever reason) and the provision_state of the node is DEPLOYING (the state after the ramdisk have POST'd back) the node will get stuck on that state.
Currently we don't have a timeout for deployments after the DEPLOYWAIT state andalso we don't offer any API that the user can use to get the node out of the DEPLOYING state.
How to reproduce:
The easiest way to reproduce it is to simulate the conductor abruptly dying by killing it with "kill -9". So:
1) Start a deployment
2) Keep looking af the provision state of the node
3) Once it goes from "wait for call-back" to "deploying", kill the ironic-conductor with: kill -9 <ir-cond PID>
Workaround:
The only way I found was to modify the database changing the provision_state of the node from "deploying" to "deploy failed" and then you can do:
$ironic node-set-
To clean the node up.
description: | updated |
description: | updated |
Changed in ironic: | |
assignee: | nobody → Tan Lin (tan-lin-good) |
Changed in ironic: | |
assignee: | Tan Lin (tan-lin-good) → Lucas Alvares Gomes (lucasagomes) |
Changed in ironic: | |
assignee: | Lucas Alvares Gomes (lucasagomes) → Tan Lin (tan-lin-good) |
Changed in ironic: | |
assignee: | Tan Lin (tan-lin-good) → Lucas Alvares Gomes (lucasagomes) |
Changed in ironic: | |
assignee: | Lucas Alvares Gomes (lucasagomes) → Tan Lin (tan-lin-good) |
Changed in ironic: | |
assignee: | Tan Lin (tan-lin-good) → Lucas Alvares Gomes (lucasagomes) |
Changed in ironic: | |
assignee: | Lucas Alvares Gomes (lucasagomes) → Jim Rollenhagen (jim-rollenhagen) |
assignee: | Jim Rollenhagen (jim-rollenhagen) → Lucas Alvares Gomes (lucasagomes) |
Changed in ironic: | |
milestone: | none → 4.0.0 |
status: | Fix Committed → Fix Released |
I believe we should have timeouts for all transient states.