race condition between delete instance(with floating ip associated) and delete floating ip

[wangpan]

reproduce steps:
1. create an instance
2. get a floating ip from pool
3. associate the ip to instance created at step 1
4. delete the instance and the ip at a very little interval

then "ERROR: Floating ip is not associated (HTTP 400) (Request-ID: req-7af58a3e-7198-42d1-afce-61e2a30d607c)" was returned, and the floating ip is still there by ubuntu@ubuntu:/opt/stack/nova$ nova floating-ip-list
+--------------+-------------+----------+------+
| Ip | Instance Id | Fixed Ip | Pool |
+--------------+-------------+----------+------+
| 172.24.4.225 | None | None | nova |
+--------------+-------------+----------+------+
and the instance is deleted successfully.

this is a race issue, so you may need to hack in the nova-api codes to reproduce it, like what I did as bellow:
diff --git a/nova/api/openstack/compute/contrib/floating_ips.py b/nova/api/openstack/compute/contrib/floating_ips.py
index bf1246c..d9cb2c2 100644
--- a/nova/api/openstack/compute/contrib/floating_ips.py
+++ b/nova/api/openstack/compute/contrib/floating_ips.py
@@ -181,11 +181,15 @@ class FloatingIPController(object):
             raise webob.exc.HTTPNotFound(explanation=msg)
         address = floating_ip['address']

         # get the associated instance object (if any)
         instance = get_instance_by_floating_ip_addr(self, context, address)

         # disassociate if associated
         if floating_ip.get('fixed_ip_id'):
+ import time
+ LOG.debug(_("xxxxxxxxxxxxxxxxx------------------------- sleep 10s start ----------xxxxxxxxxxxxxx"))
+ time.sleep(10)
             disassociate_floating_ip(self, context, instance, address)

         # release ip from project

then you can delete floating ip at first, and wait for the nova-api debug log of 'sleep 10s', delete the instance when you see this log, this bug will occurs at most of time.

reason may be that:
1. the action of delete ip find the ip is associated to a fixed ip at first(at api level where my debug log is added)
2. and then the fixed ip is released when the instance is deleted
3. so nova-network manager find this issue and raise exception.FloatingIpNotAssociated(address=floating_address)

I think the expected result is that the floating ip is deleted successfully on this condition.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/27470

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27470
Committed: http://github.com/openstack/nova/commit/a5923541e990326a9b850926e74026d5947ae142
Submitter: Jenkins
Branch: master

commit a5923541e990326a9b850926e74026d5947ae142
Author: Wangpan <email address hidden>
Date: Thu Apr 25 17:22:50 2013 +0800

    Fixes race condition of deleting floating ip

    Fixes bug #1172599
    reproduce steps:
    1. create an instance
    2. get a floating ip from pool
    3. associate the ip to instance created at step 1
    4. delete the instance and the ip at a very little interval

    reason may be that:
    1. the action of delete ip find the ip is associated to a fixed ip at
       first(at api level)
    2. and then the fixed ip is released when the instance is deleted
    3. so nova-network manager findes this issue and raises
       exception.FloatingIpNotAssociated(address=floating_address)

    Change-Id: I253d0b240451e8a728532639e9857894f6fc6870

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/27744

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/27744
Committed: http://github.com/openstack/nova/commit/726ae59b82d94e5f81e2512991776c001f22daa0
Submitter: Jenkins
Branch: stable/grizzly

commit 726ae59b82d94e5f81e2512991776c001f22daa0
Author: Wangpan <email address hidden>
Date: Thu Apr 25 17:22:50 2013 +0800

    Fixes race condition of deleting floating ip

    Fixes bug #1172599
    reproduce steps:
    1. create an instance
    2. get a floating ip from pool
    3. associate the ip to instance created at step 1
    4. delete the instance and the ip at a very little interval

    reason may be that:
    1. the action of delete ip find the ip is associated to a fixed ip at
       first(at api level)
    2. and then the fixed ip is released when the instance is deleted
    3. so nova-network manager findes this issue and raises
       exception.FloatingIpNotAssociated(address=floating_address)

    Change-Id: I253d0b240451e8a728532639e9857894f6fc6870
    (cherry picked from commit a5923541e990326a9b850926e74026d5947ae142)