Heat (icehouse) is rarely able to successfully delete stacks

[Lars Kellogg-Stedman]

I have been creating and deleting stacks frequently over the past few days, and I am become unnecessarily familar with the "heat stack-abandon" command. It appears as if Heat has an ordering problem when attempting to delete stacks that contain floating ip addresses. I am deploying using this template:

  https://github.com/larsks/heat-docker-example/blob/master/docker-server.yml

Typically, when I try to delete the stack, it ends up in the DELETE_FAILED state and logs the following error:

2014-08-30 20:52:37.683 19534 TRACE heat.engine.resource NeutronClientException: 409-{u'NeutronError': {u'message': u'Router interface for subnet 5e86ce3b-f462-47a5-bb38-ef9ea42a2734 on router 05707977-2af2-4d75-9589-717eec8c23d4 cannot be deleted, as it is required by one or more floating IPs.', u'type': u'RouterInterfaceInUseByFloatingIP', u'detail': u''}}

If I manually delete the server to which the floating ip is attached, the delete then fails because Heat is unable to delete the Docker container resources I have defined (because I have deleted the server that was hosting the resources):

2014-08-30 22:50:48.002 19534 TRACE heat.engine.resource ConnectionError: HTTPConnectionPool(host='192.168.200.44', port=2375): Max retries exceeded with url: /v1.12/containers/67997af6d0503dfb3349afd5c99ea1e2f8a8ed3c7f9010f8e7502b3d8bb174f0/kill (Caused by <class 'socket.error'>: [Errno 113] EHOSTUNREACH)

While all of these depdendencies make great sense when *creating* a stack, I'm beginning to feel that Heat is a little too strict when *deleting* a stack. There ought to be something between "heat stack-delete" and "heat stack-abandon" that will attempt to delete all stack resources and that will ignore errors. Something like "heat stack-delete --force". While "stack-abandon" is able to clear the error, it requires one to manually delete the entire resource stack, which can be time consuming and problematic if you forget to delete something and you unexpectedly run up against your Neutron security group quota.

Or perhaps a "resource-abandon" command, if not a "delete --force".

[Steve Baker (steve-stevebaker)]

Lars, this looks like a duplicate of bug #1299259 which has a suggested workaround https://bugs.launchpad.net/heat/+bug/1299259/comments/4

Can you try adding this depends_on to your template:
  docker_server_floating:
    type: "OS::Neutron::FloatingIP"
    depends_on: extrouter_inside
    properties:
      ...

This issue is that neutron's domain model doesn't map very well with their API, so there are implicit resource dependencies which show up as race conditions on stack create or stack delete. In this particular case the race is only triggered on stack delete.

We already have a bunch of workarounds for these neutron issues, but bug #1299259 hasn't been fixed yet (it will be for Juno)

Let us know if the workaround works for you, and mark this as a duplicate if so.

[Lars Kellogg-Stedman (larsks)]

The workaround seems to solve the primary problem of deletes failing because of the floating ip address. Thanks.