Add retry logic to other resource operations

[Anant Patil]

Currently, only the create operation is retried. Other operations too need to have retry logic, especially, delete and update.

[Steve Baker (steve-stevebaker)]

Retries are just a workaround for not having convergence yet. Retry on delete would be useful (especially for neutron dependencies which can't be modelled in heat)

As for update, it may be more appropriate to never retry automatically, since the user can retry manually, then wait for convergence as a proper fix.

[Anant Patil (ananta)]

 Hi Rico,
I started working on this, sorry I didn't assign it to myself.

[Rico Lin (rico-lin)]

Don't worry, u can have it back :)

[Anant Patil (ananta)]

I think retry is still needed in convergence. When the reality is not
matched, the resources try to converge and there will be retries in this
process... and if the retries also fail, the resource is marked as
failed.

IMO, every decision to converge a resource will result in retries of
convergence action. We will have to keep the retry logic around.

[Anant Patil (ananta)]

Thanks Rico!

[Satyanarayana Patibandla (satya-patibandla)]

When we try to delete multiple heat stacks using a single heat delete command, few stack deletes are failing due to failing neutron floating IP resource deletion. if I delete failed stacks again they are getting deleted properly. And also If I delete heat stacks one after the other it is working fine. if there is resource delete retry logic inside the heat itself, the deletion of multiple stacks with a single heat delete command will work properly.

[Anant Patil (ananta)]

There is a retry logic in heat for deleting a resource: https://github.com/openstack/heat/blob/master/heat/engine/resource.py#L1721

Probably the resource deletion is not reflected for the dependent resource to proceed for deletion. Please share the output in log files that you get when attempt to delete a resource fails.