Rapidly removing a floating ip can leave behind nat rules

Bug #1092762 reported by Vish Ishaya
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vish Ishaya
Folsom
Fix Released
High
Vish Ishaya

Bug Description

Associating and disassociating a floating ip very rapidly can lead to iptables rules being left behind. This is because due to lag in the api host or network host that passes along the request, the removal of iptables rules can hit before the addition of rules.

example:
$ nova list
+--------------------------------------+------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+------------------+
| e2212fd7-227d-447f-aea1-71e4c682765a | foo1 | ACTIVE | private=10.0.0.3 |
+--------------------------------------+------+--------+------------------+

$ (nova add-floating-ip foo1 10.0.0.201 &); sleep 1.0; (nova remove-floating-ip foo1 10.0.0.201 &);

$ nova list
+--------------------------------------+------+--------+------------------+
| ID | Name | Status | Networks |
+--------------------------------------+------+--------+------------------+
| e2212fd7-227d-447f-aea1-71e4c682765a | foo1 | ACTIVE | private=10.0.0.3 |
+--------------------------------------+------+--------+------------------+

<on host with foo1>

$ sudo iptables -t nat -L -n -v | grep 10.0.0.3
    0 0 DNAT all -- * * 0.0.0.0/0 10.0.0.201 to:10.0.0.3
    0 0 DNAT all -- * * 0.0.0.0/0 10.0.0.201 to:10.0.0.3
    0 0 SNAT all -- * ext0 10.0.0.3 0.0.0.0/0 to:10.0.0.201

Changed in nova:
importance: Undecided → High
status: New → In Progress
assignee: nobody → Vish Ishaya (vishvananda)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/18525

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/18526

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/18572

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/18572
Committed: http://github.com/openstack/nova/commit/3b9f08072abe9f92d292e9fa5998c62a766b01f3
Submitter: Jenkins
Branch: master

commit 3b9f08072abe9f92d292e9fa5998c62a766b01f3
Author: Vishvananda Ishaya <email address hidden>
Date: Fri Dec 21 16:04:16 2012 -0800

    Ignore double messages to associate the same ip

    Associating an ip removes the old association and adds a new one.
    If we are already associated to the target ip, then we can avoid
    sending extra messages and making db calls. The current possibility
    of a double send contributes to bug 1092762.

    Change-Id: I5a40177fcd05b150f39e3144c1d521bd979b358b

Changed in nova:
status: In Progress → Fix Committed
Mark McLoughlin (markmc)
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/18525
Committed: http://github.com/openstack/nova/commit/881a93473c32a7c7e23a8e6dcede8394053408c6
Submitter: Jenkins
Branch: master

commit 881a93473c32a7c7e23a8e6dcede8394053408c6
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Dec 20 20:13:37 2012 -0800

    Eliminate race conditions in floating association

    This makes associating and disassociating floating ips atomic and
    idempotent. This means multiple concurrent messages will not leave
    behind iptables rules and concurrent request will not cause odd
    failures.

    Fixes bug 1092762 and bug 1092761.

    Change-Id: Idbcad6c1d2a3d4881cf7180b848ed3844fac4054

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Mark McLoughlin (markmc)
tags: removed: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/18526
Committed: http://github.com/openstack/nova/commit/5a6681222999873f0df9816125fe9888498d91c2
Submitter: Jenkins
Branch: stable/folsom

commit 5a6681222999873f0df9816125fe9888498d91c2
Author: Vishvananda Ishaya <email address hidden>
Date: Thu Dec 20 20:13:37 2012 -0800

    Eliminate race conditions in floating association

    This makes associating and disassociating floating ips atomic and
    idempotent. This means multiple concurrent messages will not leave
    behind iptables rules and concurrent request will not cause odd
    failures.

    Fixes bug 1092762 and bug 1092761.

    Change-Id: Idbcad6c1d2a3d4881cf7180b848ed3844fac4054
    (cherry picked from commit 881a93473c32a7c7e23a8e6dcede8394053408c6)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-2 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.