Multiple floating IPs from the same external network are associated to one port when commands are executed at the same time

[Lujin Luo]

I have three controller nodes and the Neutron servers on these controllers are set behind Pacemaker and HAProxy to realize active/active HA using DevStack. MariaDB Galera cluster is used as my database backend. I am using the latest codes.

If I have multiple commands to create floating IPs and associate them to the same port at the same time, all of the commands would return success and end up with multiple floating IPs from the same external network associated to the same port.

How to reproduce:

Step 1: Create a network
$ neutron net-create net1

Step 2: Create a subnet on the network
$ neutron subnet-create --name subnet1 net1 192.168.100.0/24

Step 3: Create a port on the network
$ neutron port-create net1

Step 4: Create a router
$ neutron router-create router-floatingip-test

Step 5: Add the subnet as its interface
$ neutron router-interface-add router-floatingip-test subnet1

Step 5: Create an external network
$ neutron net-create ext-net --router:external True

Step 6: Add a subnet on the external network
$ neutron subnet-create --name ext-subnet ext-net 192.168.122.0/24

Step 7: Set the external network as the router's default gateway
$ neutron router-gateway-set router-floatingip-test ext-net

Step 8: Run the three commands at the same time to create floating IPs
On controller1:
$ neutron floatingip-create ext-net --port-id b53d0826-53c4-427b-81b2-3ab6cb0f4511

On controller2:
$ neutron floatingip-create ext-net --port-id b53d0826-53c4-427b-81b2-3ab6cb0f4511

On controller3:
$ neutron floatingip-create ext-net --port-id b53d0826-53c4-427b-81b2-3ab6cb0f4511

where, port_id b53d0826-53c4-427b-81b2-3ab6cb0f4511 is the port we created in Step 3.

The result would be three floating IPs associated to the same port, as shown in http://paste.openstack.org/show/483691/

The expected error message (say, we run the second command after the first one succeeds) would be
Cannot associate floating IP 192.168.122.20 (bd4d47a5-45c1-48e1-a48a-aef08039a955) with port b53d0826-53c4-427b-81b2-3ab6cb0f4511 using fixed IP 192.168.100.3, as that fixed IP already has a floating IP on external network 920ee0f3-3db8-4005-8d29-0be474947186.
Since one port with one fixed_ip is not allowed to have multiple floating IPs from the same external network.

In the above procedure, I set port_id when creating these three floating IPs. Same bug occurred when I updated three existing floating IPs to be associated with the same port at the same time.

I assume this bug happens because multiple APIs are executed concurrently and the validation check on every API succeeds [1].

[1] https://github.com/openstack/neutron/blob/master/neutron/db/l3_db.py#L915

--------------------Update on 2016/9/8-------------------------------------
The clearer bug description should be to mitigate one fixed_ip_address on one port to be associated with multiple floating_ip_address(es) from the same floating_network_id.

This implies the following 2 use cases are allowed/supported in Neutron:
1. one port_id with different fixed_ip_address(es) can associate with floating_ip_address(es) from the same floating_network_id [2]

2. same fixed_ip_address can associate to same floating_network_id, as long as they are on different ports, because different internal networks could be using the same IP ranges

[2] https://bugs.launchpad.net/neutron/+bug/1057844

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/276018

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/276018
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a9c3b7ef08a34b27b63a12cfbd5798d13b083fc1
Submitter: Jenkins
Branch: master

commit a9c3b7ef08a34b27b63a12cfbd5798d13b083fc1
Author: Lujin Luo <email address hidden>
Date: Wed Feb 3 17:56:53 2016 +0900

    Add a composite unique key to floatingip table in Neutron database

    This patch set avoids associating multiple floating IPs to one fixed_
    ip_address on one internal port when commands are executed concurrently
    by adding a composite unique key between floating_network_id, fixed_port
    _id and fixed_ip_address in floatingips table in Neutron database.

    This implies the following 2 use cases are allowed/supported in Neutron:
    1. one port_id with different fixed_ip_address(es) can associate with
    floating_ip_address(es) from the same floating_network_id [1]

    2. same fixed_ip_address can associate to same floating_network_id, as
    long as they are on different ports, because different internal networks
    could be using the same IP ranges.

    [1] https://bugs.launchpad.net/neutron/+bug/1057844

    Change-Id: Ie8f3ec1b23c14f36992886510c3114cf956769d4
    Closes-Bug: #1534445

[OpenStack Infra (hudson-openstack)]

This issue was fixed in the openstack/neutron 9.0.0.0rc1 release candidate.