Retry of nested stack delete has race

[Zane Bitter]

When updating a nested stack that is e.g. in the UPDATE_FAILED state, we check the timestamp when polling the child status to avoid a race where we may see the status before the child stack becomes UPDATE_IN_PROGRESS again.

There is no such protection for a nested stack in the DELETE_FAILED state. Trying to delete the parent resource again will likely always fail with the same error as last time, since it will see the status before the child stack becomes DELETE_IN_PROGRESS. (A subsequent attempt should work, if the stack was in fact deleted, since the parent will get NotFound when looking for the child status.)

[Al Bailey (albailey1974)]

delete_stack used to always use cast, and now it looks in the code like it always uses call.

I believe the problem is here:
   https://github.com/openstack/heat/blob/master/heat/engine/resources/stack_resource.py#L535

(needs a return statement. Same for the abandon_stack 2 lines above)

  return self.rpc_client().delete_stack(self.context, stack_identity, cast=False)

otherwise delete_nested will immediately proceed to the check_delete method (which detects FAILED) may execute before the handle_delete is able to update the state for the nested stack.

[Zane Bitter (zaneb)]

No, it needs to return some data in a similar format to what handle_update() does:

https://github.com/openstack/heat/blob/master/heat/engine/resources/stack_resource.py#L484-L486

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/508443

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/509769

[OpenStack Infra (hudson-openstack)]

Related fix proposed to branch: master
Review: https://review.openstack.org/510359

[Ethan Lynn (ethanlynn)]

Met this exactly problem at my env. Wondering is there a way to fix it?