[DOC] Restart of sahara-engine can leave cluster in "Starting" state

[Matthew Farrellee]

Theme - Savanna should be tolerant of its own faults

Cluster creation happens in an eventlet running api.py:_cluster_creation_job within the savanna-api process. If the savanna-api process is restarted during cluster creation[0], the cluster creation process is not restarted.

The result is all clusters that were "Starting" at the time of the restart will always be "Starting", i.e. never transition to "Active".

Expectation is that the startup process will be resumed once the savanna-api process is restarted.

[0] A long lived operation, because it waits for all instances to be up and accessible

[Sergey Lukjanov (slukjanov)]

We are working at the next phase with pluggable provisioning support and due to the supporting plugins that will provide an ability to deploy management consoles cluster startup process will be very complicated process. Generally I think that we should support resuming cluster creation after savanna-api restart, but it's a very good question - how it should work?

I think that blueprint should be created for such problem with several ways to solve it.

[Sergey Lukjanov (slukjanov)]

BTW it is an extremely good point and it would be really cool to solve this problem as well as the problem with allocated resource management.

[Andrew Lazarev (alazarev)]

I think this one has pretty simple solution - move non-Active cluster to 'Error' after some period of time. This solution is not ideal, but much better than having cluster in 'Starting' forever.

[Sergey Lukjanov (slukjanov)]

This bug is > 180 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Sergey Reshetnyak (sreshetniak)]

This bug is > 60 days without activity. We are unsetting assignee and milestone and setting status to Incomplete in order to allow its expiry in 60 days.

If the bug is still valid, then update the bug status.

[Luigi Toscano (ltoscano)]

This is still true (try to restart sahara-engine during cluster creation, the creation is not resumed, whether it was in Progress, Configuring or Starting).
On the other side, it is possible to configure cleanup_time_for_incomplete_clusters to automatically cleanup clusters in non-final states, but it is not enabled by default. This bug is about the default behavior and I think then that it's not solved.

[Vitalii Gridnev (vgridnev)]

It's hard to predict what value should be used by default. I think that we should document that feature well. adding docs tags and moving to Medium prio + newton-3

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/340422

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/340422
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=c90a1da596411743c8a11310112b420dbc1380ef
Submitter: Jenkins
Branch: master

commit c90a1da596411743c8a11310112b420dbc1380ef
Author: Michael Ionkin <email address hidden>
Date: Mon Jul 11 18:00:16 2016 +0300

    [DOC] Cleanup time for incomplete clusters

    Cleanup time for incomplete clusters feature documented.
    closes-bug: 1185909

    Change-Id: I785b25b30e88cf0604236d34aaf3170b990b21d3

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/sahara 5.0.0.0b3 development milestone.