L3 agent can't handle updates that change floating ip id

[Carl Baldwin]

The problem occurs when a network update comes along where a new floating ip id carries the same (reused) IP address as an old floating IP. In short, same address, different floating ip id. We've seen this occur in testing where the floating ip free pool has gotten small and creates/deletes come quickly.

What happens is the agent skips calling "ip addr add" for the address since the address already appears. It then calls "ip addr del" to remove the address from the qrouter's gateway interface. It shouldn't have done this and the floating ip is left in a non-working state.

Later, when the floating ip is disassociated from the port, the agent attempts to remove the address from the device which results in an exception which is caught above. The exception prevents the iptables code from removing the DNAT address for the floating ip.

2013-07-23 09:20:06.094 3109 DEBUG quantum.agent.linux.utils [-] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2b75022a-3721-443f-af99-ec648819d080', 'ip', '-4', 'addr', 'del', '15.184.103.155/32', 'dev', 'qg-c847c5a7-62'] execute /usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py:42
2013-07-23 09:20:06.179 3109 DEBUG quantum.agent.linux.utils [-]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2b75022a-3721-443f-af99-ec648819d080', 'ip', '-4', 'addr', 'del', '15.184.103.155/32', 'dev', 'qg-c847c5a7-62']
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: Cannot assign requested address\n' execute /usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py:59

The DNAT entries in the iptables stay in a bad state from this point on sometimes preventing other floating ip addresses from being attached to the same instance.

I have a fix for this that is currently in testing. Will submit for review when it is ready.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/40797

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/41584

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/41584
Committed: http://github.com/openstack/neutron/commit/9382ee659212285a203550cf60476dd146d27a29
Submitter: Jenkins
Branch: master

commit 9382ee659212285a203550cf60476dd146d27a29
Author: Carl Baldwin <email address hidden>
Date: Tue Aug 13 00:11:29 2013 +0000

    Refactor configuring of floating ips on a router.

    This approach to configuring floating ips is stateless and idempotent.
    This allows it to handle corner cases, such as reusing a floating ip
    address with a different floating ip id in a way that is easier to
    understand.

    The concept is to wipe the floating ips clean and rebuild them each
    time with the following optimizations. To avoid bad performance in
    manipulating iptables, it is called in the context of a call to
    defer_apply_on. To avoid a disruption in network flow a set
    difference is use to determine the set of addresses that no longer
    belong on the inteface rather than removing them all blindly.

    Change-Id: I0cfb58d487b1925e0a0db2a701c5ea3c56a0b2b5
    Fixes: Bug #1209011

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/42412

[Carl Baldwin (carl-baldwin)]

Mark, note that the fix committed status is wrong since the initial fix was reverted.

[Carl Baldwin (carl-baldwin)]

Please reopen. The fix has not been released. The first fix was reverted due to a gate breakage and the follow-on fix has suffered from lack of reviews.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/53689

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/42412
Committed: http://github.com/openstack/neutron/commit/a65188fab01f29d095031abbc8d1d194548cd8be
Submitter: Jenkins
Branch: master

commit a65188fab01f29d095031abbc8d1d194548cd8be
Author: Carl Baldwin <email address hidden>
Date: Fri Sep 27 04:04:31 2013 +0000

    Refactor configuring of floating ips on a router

    This approach to configuring floating ips is stateless and idempotent.
    This allows it to handle corner cases, such as reusing a floating ip
    address with a different floating ip id in a way that is easier to
    understand.

    The concept is to wipe the floating ips clean and rebuild them each
    time with the following optimizations. To avoid bad performance in
    manipulating iptables, it is called in the context of a call to
    defer_apply_on. To avoid a disruption in network flow a set
    difference is use to determine the set of addresses that no longer
    belong on the inteface rather than removing them all blindly.

    Change-Id: I98aacbbb52b35688036990961d02e0b273504a77
    Fixes: Bug #1209011

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/53689
Committed: http://github.com/openstack/neutron/commit/38211a2f4167ca12850f6e771899fffce6da72f2
Submitter: Jenkins
Branch: stable/havana

commit 38211a2f4167ca12850f6e771899fffce6da72f2
Author: Carl Baldwin <email address hidden>
Date: Fri Sep 27 04:04:31 2013 +0000

    Refactor configuring of floating ips on a router

    This approach to configuring floating ips is stateless and idempotent.
    This allows it to handle corner cases, such as reusing a floating ip
    address with a different floating ip id in a way that is easier to
    understand.

    The concept is to wipe the floating ips clean and rebuild them each
    time with the following optimizations. To avoid bad performance in
    manipulating iptables, it is called in the context of a call to
    defer_apply_on. To avoid a disruption in network flow a set
    difference is use to determine the set of addresses that no longer
    belong on the inteface rather than removing them all blindly.

    Change-Id: I0cfb58d487b1925e0a0db2a701c5ea3c56a0b2b5
    Fixes: Bug #1209011