Node stuck on DEPLOYING (potentially all *ING) state(s)

[Lucas Alvares Gomes]

If a conductor deploying a node dies due some OOM Killer (or whatever reason) and the provision_state of the node is DEPLOYING (the state after the ramdisk have POST'd back) the node will get stuck on that state.

Currently we don't have a timeout for deployments after the DEPLOYWAIT state andalso we don't offer any API that the user can use to get the node out of the DEPLOYING state.

How to reproduce:

The easiest way to reproduce it is to simulate the conductor abruptly dying by killing it with "kill -9". So:

1) Start a deployment
2) Keep looking af the provision state of the node
3) Once it goes from "wait for call-back" to "deploying", kill the ironic-conductor with: kill -9 <ir-cond PID>

Workaround:

The only way I found was to modify the database changing the provision_state of the node from "deploying" to "deploy failed" and then you can do:

$ironic node-set-provision-state <node uuid> deleted

To clean the node up.

[Dmitry Tantsur (divius)]

I believe we should have timeouts for all transient states.

[Tan Lin (tan-lin-good)]

It's reasonable to have timeouts for all transient states, I will add things like checkout_deploying_timeouts/checkout_delete_timeouts/checkout_clean_timeouts

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/188688

[Tan Lin (tan-lin-good)]

Thing are getting interesting. Once conductor restart, it will clear all locks on the host during its init process by running "self.dbapi.clear_node_reservations_for_conductor(self.host)"

So we have two cases now,:
1. Break by restart of conductor(reservation=None)
2. Break by something else so reservation is still fake-reserv

Case one is similar like what Ironic did in _check_deploy_timeouts.
In case two, we have to release lock from a reserved node maybe named "force_release_resources" like release_resource() in task_manager, but I am not sure this should be allowed?

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/189587

[Lucas Alvares Gomes (lucasagomes)]

Hi Tan,

Any news on this?

[Tan Lin (tan-lin-good)]

I proposed two patches to review:
https://review.openstack.org/#/c/188688/ Break by restart of conductor(reservation=None)
https://review.openstack.org/#/c/189587/ Break by something else so reservation is still hold

[Lucas Alvares Gomes (lucasagomes)]

Hi Tan,

Please also take a look at https://review.openstack.org/#/c/194132/

[Jim Rollenhagen (jim-rollenhagen)]

So this bug is actually invalid for the agent driver -- the node is not locked for much of the deployment and can actually recover just fine from a conductor crash (heartbeats will just be routed to another conductor). Raising another bug around how https://review.openstack.org/#/c/194132/ breaks this.

[Jim Rollenhagen (jim-rollenhagen)]

Maybe I'll just keep the bug discussion here. tl;dr 194132 assumes that anything in DEPLOYING should also be locked, which is an invalid assumption with the agent driver.

[Ruby Loo (rloo)]

I'm a bit confused. Lucas submitted this patch to "Periodically checks the status of nodes in DEPLOYING state" : https://review.openstack.org/#/c/197141/

[Jim Rollenhagen (jim-rollenhagen)]

So I didn't really get around to this today, but poked around a bit. Fixing the agent driver to properly use DEPLOYWAIT is likely the right thing to do, however the agent relies on the DEPLOYING state to know whether it should start or finish the deploy.

I'd love for commands to signal back to a specific endpoint or something, rather than just riding the heartbeat for everything we do. However this is a lot of work and will have backwards compat concerns etc.

Short term, maybe we can fix 194132 by moving the block to fail the deploys above where the conductor releases the lock? Would need to play with it a bit.

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/200153

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/200153
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f1929f0155e25c83bafe64c3d235880fc486f323
Submitter: Jenkins
Branch: master

commit f1929f0155e25c83bafe64c3d235880fc486f323
Author: Jim Rollenhagen <email address hidden>
Date: Thu Jul 9 08:39:04 2015 -0700

    Use DEPLOYWAIT while waiting for agent to write image

    There is an assumption that anything in DEPLOYING state is locked; the
    agent driver breaks that assumption. Fix it by using DEPLOYWAIT while
    waiting for the agent to write an image.

    Change-Id: I4957bd9608b1bc92177e69efef66ecb951181de1
    Related-Bug: #1461937

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/197141
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=e2a145e27d874d6c46a63340abf3510fc3169fbe
Submitter: Jenkins
Branch: master

commit e2a145e27d874d6c46a63340abf3510fc3169fbe
Author: Lucas Alvares Gomes <email address hidden>
Date: Tue Jun 30 13:05:01 2015 +0100

    Periodically checks the status of nodes in DEPLOYING state

    Periodically checks the nodes in DEPLOYING and the state of the conductor
    deploying them. If we find out that a conductor that was provisioning
    the node has died we then break release the node and gracefully mark
    the deployment as failed.

    A new method called "get_offline_conductors" was added to the database
    api which return a list of conductor hostnames that are offline.

    Closes-Bug: #1461937
    Change-Id: I293d3ff18e949b305e6404b8d3ceb75e29b88356

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tan Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/188688

[OpenStack Infra (hudson-openstack)]

Change abandoned by Tan Lin (<email address hidden>) on branch: master
Review: https://review.openstack.org/189587

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/349971

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/350439