Failed to delete stack with Sahara cluster and template

[Andrew Lazarev]

Steps to reproduce:
1. Create stack with 1 nodegroup template, 1 cluster template (based on ng tmpl), and cluster (based on tmpl). It will look something like:

resources:
  fake_ng_tmpl:
    type: OS::Sahara::NodeGroupTemplate
    properties:
      ...
  fake_ct_tmpl:
    type: OS::Sahara::ClusterTemplate
    properties:
      node_groups: [{"count": 1, name: 'fake-ng', "node_group_template_id": { get_resource: fake_ng_tmpl }}]
      ...
  fake_cluster:
    type: OS::Sahara::Cluster
    properties:
      cluster_template_id: { get_resource: fake_ct_tmpl }
      ...

2. Try to delete stack

Stacktrace:
2014-12-08 15:58:39.476 INFO heat.engine.resource [-] DELETE: SaharaClusterTemplate "fake_ct_tmpl" [5f3414c9-1323-421e-a733-e1d98066c59b] Stack "f8" [90e3097b-e915-4514-8062-43f4b758fd3d]
2014-12-08 15:58:39.476 TRACE heat.engine.resource Traceback (most recent call last):
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 462, in _action_recorder
2014-12-08 15:58:39.476 TRACE heat.engine.resource yield
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 885, in delete
2014-12-08 15:58:39.476 TRACE heat.engine.resource yield self.action_handler_task(action, *action_args)
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/scheduler.py", line 295, in wrapper
2014-12-08 15:58:39.476 TRACE heat.engine.resource step = next(subtask)
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resource.py", line 503, in action_handler_task
2014-12-08 15:58:39.476 TRACE heat.engine.resource handler_data = handler(*args)
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/resources/sahara_templates.py", line 360, in handle_delete
2014-12-08 15:58:39.476 TRACE heat.engine.resource self.client_plugin().ignore_not_found(ex)
2014-12-08 15:58:39.476 TRACE heat.engine.resource File "/opt/stack/heat/heat/engine/clients/client_plugin.py", line 90, in ignore_not_found
2014-12-08 15:58:39.476 TRACE heat.engine.resource raise ex
2014-12-08 15:58:39.476 TRACE heat.engine.resource APIException: Cluster template 5f3414c9-1323-421e-a733-e1d98066c59b in use by fake

[Andrew Lazarev (alazarev)]

Exact template attached.

[Pavlo Shchelokovskyy (pshchelo)]

So, looks like cluster delete might be quite lengthy operation, so we need to implement check_delete_complete for that resource. A question right away then - can the same be true for template resources (I suspect no, as those are just some entries in internal Sahara DB, but just to be sure..)?

[Andrew Lazarev (alazarev)]

Sahara has 2 types of objects that are deleted asynchronously: clusters and job executions. Since job executions are not supported by heat yet, clusters are the only objects that should be checked.

[Andrew Lazarev (alazarev)]

Note, to make attached template work it is required to apply fixes of https://bugs.launchpad.net/sahara/+bug/1399052 and https://bugs.launchpad.net/sahara/+bug/1399056

[Sergey Kraynev (skraynev)]

It's reproduced for me, and I suppose, that fix suggested by Pavlo should help with this issue.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/141059

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/141059
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=122bb66675d6d6d3691769807ea93a6d279768c2
Submitter: Jenkins
Branch: master

commit 122bb66675d6d6d3691769807ea93a6d279768c2
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Thu Dec 11 14:13:53 2014 +0000

    Wait for Sahara Cluster to be deleted

    Currently the stack with cluster resource depending on cluster template
    can not be deleted as the deletion of the cluster template fails since
    actually the dependent cluster is not completely deleted yet.

    This patch implements check_delete_complete for OS::Sahara::Cluster resource,
    allowing the resource to wait until the actual Sahara cluster is deleted
    before proceeding along dependency graph.

    Change-Id: I9bf3192b34081f045f8165a3b958a7f9a6418ff7
    Closes-Bug: #1400540