The problem occurs when a network update comes along where a new floating ip id carries the same (reused) IP address as an old floating IP. In short, same address, different floating ip id. We've seen this occur in testing where the floating ip free pool has gotten small and creates/deletes come quickly.
What happens is the agent skips calling "ip addr add" for the address since the address already appears. It then calls "ip addr del" to remove the address from the qrouter's gateway interface. It shouldn't have done this and the floating ip is left in a non-working state.
Later, when the floating ip is disassociated from the port, the agent attempts to remove the address from the device which results in an exception which is caught above. The exception prevents the iptables code from removing the DNAT address for the floating ip.
2013-07-23 09:20:06.094 3109 DEBUG quantum.agent.linux.utils [-] Running command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2b75022a-3721-443f-af99-ec648819d080', 'ip', '-4', 'addr', 'del', '15.184.103.155/32', 'dev', 'qg-c847c5a7-62'] execute /usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py:42
2013-07-23 09:20:06.179 3109 DEBUG quantum.agent.linux.utils [-]
Command: ['sudo', 'quantum-rootwrap', '/etc/quantum/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-2b75022a-3721-443f-af99-ec648819d080', 'ip', '-4', 'addr', 'del', '15.184.103.155/32', 'dev', 'qg-c847c5a7-62']
Exit code: 2
Stdout: ''
Stderr: 'RTNETLINK answers: Cannot assign requested address\n' execute /usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py:59
The DNAT entries in the iptables stay in a bad state from this point on sometimes preventing other floating ip addresses from being attached to the same instance.
I have a fix for this that is currently in testing. Will submit for review when it is ready.
Fix proposed to branch: master /review. openstack. org/40797
Review: https:/