Nodes left in an inconsistent state due to lack of free conductor workers to start the deployment

[Lucas Alvares Gomes]

Before spawning the greenthread that will actually do the work for provisioning a node Ironic first set the provision_state of the node to DEPLOYING and target_provision_state to DEPLOYED to expose that the work is in progress[1], but if there's no free workers to actually deploy that machine the deployment fails and the nodes are left in an incosistent state where it's impossible to try to hit deploy again nor delete the node.

LOGS: http://paste.openstack.org/show/84394/

[1] https://github.com/openstack/ironic/blob/master/ironic/conductor/manager.py#L386-L389

== How to reproduce ==

Set the config option workers_pool_size to 1 (1 for the dedicated greenthread running the keepalive)

[conductor]

# The size of the workers greenthread pool. (integer value)
workers_pool_size=1

Now try to deploy a node.

== More ==

Something similar will happen when you try to delete a node and there's no available free workers, Ironic will set the provision_state to DELETING and target_provision_state to DELETED and the operation will fail, after that the node is now in an incosistent state where it can't be deleted anymore because Ironic thinks that there's already an operation running to delete that node.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100957

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/100958

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100957
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=ebf09d96139531ef60843834beee4bb83b99b1de
Submitter: Jenkins
Branch: master

commit ebf09d96139531ef60843834beee4bb83b99b1de
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jun 18 16:20:20 2014 +0100

    Add set_spawn_error_hook to TaskManager

    This patch is adding a way to create a hook on the TaskManager that gets
    called upon an exception being raised. The task.spawn_after() do not
    raise any exception making it impossible to add some logic around it in
    case something goes bad with the method being executed by the greenthread.

    Partial-Bug: #1331494
    Change-Id: I5755df359a6e8678b64d4d59c25a9192f575b13d

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/100958
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=8b554deed62d86d19d7459ab1ac36cf2dc27745a
Submitter: Jenkins
Branch: master

commit 8b554deed62d86d19d7459ab1ac36cf2dc27745a
Author: Lucas Alvares Gomes <email address hidden>
Date: Wed Jun 18 17:19:37 2014 +0100

    Fix nodes left in an incosistent state if no workers

    This patch is fixing the problem of leaving the nodes in an inconsistent
    state if there's no free conductor workers available to deploy or the
    tear down a node, the patch is using the set_spawn_error_hook() method
    of TaskManager to run some custom code that will rollback the nodes
    to the previous provision_state and target_provision_state in case
    NoFreeConductorWorker is raised.

    Closes-Bug: #1331494
    Change-Id: I5d6e8e2c69cbdf1f9abe169afe617aa79783e57d