Node stuck on DELETING/CLEANING state(s)

[Zhenguo Niu]

When a conductor managing a node dies abruptly mid cleaing/deleting, the
node will get stuck in the CLEANING/DELETING state.

[Jay Faulkner (jason-oldos)]

Logic in the conductor should handle this in normal failover cases; can you provide logs around when this happens so we can more directly pinpoint the bug?

Until then, marking this as incomplete.

[Ruby Loo (rloo)]

This bug has been opened for over a year and there isn't enough information here, to know what to do. I'm going to set the status to "invalid", feel free to change it if you have more information. Thanks.

[Dmitry Tantsur (divius)]

Ruby, Jay, we only clean up DEPLOYING state on start up, there is no "normal failover" for CLEANING or DELETING.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/349971
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=2921fe685d8f096717f8795494c1032025407fe4
Submitter: Zuul
Branch: master

commit 2921fe685d8f096717f8795494c1032025407fe4
Author: Zhenguo Niu <email address hidden>
Date: Tue Aug 2 20:24:54 2016 +0800

    Clean nodes stuck in CLEANING state when ir-cond restarts

    When a conductor managing a node dies abruptly mid cleaing, the
    node will get stuck in the CLEANING state.

    This also moves _start_service() before creating CLEANING nodes
    in tests. Finally, it adds autospec to a few places where the tests
    fail in a mysterious way otherwise.

    Change-Id: Ia7bce4dff57569707de4fcf3002eac241a5aa85b
    Co-Authored-By: Dmitry Tantsur <email address hidden>
    Partial-Bug: #1651092

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/queens
Review: https://review.openstack.org/545893

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/545893
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=f132e1b761ed5fb9bf8b51b4d58e787551e36746
Submitter: Zuul
Branch: stable/queens

commit f132e1b761ed5fb9bf8b51b4d58e787551e36746
Author: Zhenguo Niu <email address hidden>
Date: Tue Aug 2 20:24:54 2016 +0800

    Clean nodes stuck in CLEANING state when ir-cond restarts

    When a conductor managing a node dies abruptly mid cleaing, the
    node will get stuck in the CLEANING state.

    This also moves _start_service() before creating CLEANING nodes
    in tests. Finally, it adds autospec to a few places where the tests
    fail in a mysterious way otherwise.

    Change-Id: Ia7bce4dff57569707de4fcf3002eac241a5aa85b
    Co-Authored-By: Dmitry Tantsur <email address hidden>
    Partial-Bug: #1651092
    (cherry picked from commit 2921fe685d8f096717f8795494c1032025407fe4)

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/pike
Review: https://review.openstack.org/546083

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/546083
Committed: https://git.openstack.org/cgit/openstack/ironic/commit/?id=8ad8c874d208e2c80be05bc64afe67d9a9c7a9ec
Submitter: Zuul
Branch: stable/pike

commit 8ad8c874d208e2c80be05bc64afe67d9a9c7a9ec
Author: Zhenguo Niu <email address hidden>
Date: Tue Aug 2 20:24:54 2016 +0800

    Clean nodes stuck in CLEANING state when ir-cond restarts

    When a conductor managing a node dies abruptly mid cleaing, the
    node will get stuck in the CLEANING state.

    This also moves _start_service() before creating CLEANING nodes
    in tests. Finally, it adds autospec to a few places where the tests
    fail in a mysterious way otherwise.

    Change-Id: Ia7bce4dff57569707de4fcf3002eac241a5aa85b
    Co-Authored-By: Dmitry Tantsur <email address hidden>
    Partial-Bug: #1651092
    (cherry picked from commit 2921fe685d8f096717f8795494c1032025407fe4)

[OpenStack Infra (hudson-openstack)]

Change abandoned by Zhenguo Niu (<email address hidden>) on branch: master
Review: https://review.opendev.org/350439

[Jay Faulkner (jason-oldos)]

Please update the bug if it is still an issue with Ironic now.