heat stack-delete not removing floating ip and subsequently failing

[Harry Rybacki]

Discovered while running a tripleo-ci ping test script on top of a newton deployment,
which consists of an overcloud populated by a heat stack (which created an instance, floatingips, router...).

The heat stack-delete failed on Newton each time. [1]
By digging into the heat engine and neutron server logs, we can see that heat asks to delete the floatingips and then router interface directly after[2].
But if you look at the neutron server log [3], it shows that the request for deleting the floating is done few seconds after.
The remove_router_interface occurred at 14:12:52.594 and the request for the deleting the floating ip at 2016-06-22 14:12:54.510

[1] https://paste.fedoraproject.org/383244/66039931/
[2] https://paste.fedoraproject.org/383279/66072051/
[3] https://paste.fedoraproject.org/383325/10212146/

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/333055

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/333055
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a3860b1162d4beba9fabb58862f41cf24ca742e4
Submitter: Jenkins
Branch: master

commit a3860b1162d4beba9fabb58862f41cf24ca742e4
Author: Steve Baker <email address hidden>
Date: Thu Jun 23 10:37:22 2016 +1200

    Wait for 404 on floatingip delete

    Currently the FloatingIP resource assumes that delete is an atomic
    operation, but there have been reports of downstream dependency
    deletes failing because the floatingip still exists.

    This change triggers the standard neutron resource
    check_delete_complete 404 polling to ensure the floatingip really is
    gone.

    Change-Id: I9be34ef8b64a79d3211841cb7a7acd945f1f107c
    Closes-Bug: #1595270

[Doug Hellmann (doug-hellmann)]

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.