Orphan floating ip's created via rapid delete/assign/remove operations

[Patrick Crews]

It is possible to create 'orphan' floating ip's (at least in devstack testing) through a sequence of:
delete vip
assign vip
remove vip

API calls + timestamps from a test run:
26101:[2014-11-14 17:18:57,446] mahmachine/INFO/stdout: 0x7f8833395250: delete floating ip id: 14
26145:[2014-11-14 17:18:58,237] mahmachine/INFO/stdout: 0x7f88333e8810: assign floating ip: 172.24.4.14 || d4545f39-6a5c-40e3-99f4-f72c22d56fc7
27333:[2014-11-14 17:19:25,144] mahmachine/INFO/stdout: 0x7f88333e8810: remove floating ip: 172.24.4.14 || d4545f39-6a5c-40e3-99f4-f72c22d56fc7

This results in floating ip addresses that are still listed as attached to an instance, yet are not owned (and are not removable) by the instance's owner.

In the database, the fixed_ip_id is not null (the server id), yet the project id is:
the 'host' column may or may not be populated, but the cause and effect appear to be the same regardless of this.

select id, address, fixed_ip_id, project_id, host from floating_ips where project_id IS NULL and fixed_ip_id IS NOT NULL;
+----+-------------+-------------+------------+-------------+
| id | address | fixed_ip_id | project_id | host |
+----+-------------+-------------+------------+-------------+
| 2 | 172.24.4.2 | 4 | NULL | NULL |
| 7 | 172.24.4.7 | 4 | NULL | mahmachine |
| 11 | 172.24.4.11 | 4 | NULL | mahmachine |
| 6 | 172.24.4.6 | 7 | NULL | mahmachine |
| 15 | 172.24.4.15 | 7 | NULL | mahmachine |
| 3 | 172.24.4.3 | 8 | NULL | mahmachine |
| 14 | 172.24.4.14 | 10 | NULL | NULL |
+----+-------------+-------------+------------+-------------+

[Patrick Crews (patrick-crews)]

nova behavior with such floating ips:

pcrews@mahmachine:~/git/mahrepo$ nova list
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------------------------------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------------------------------------------------------+
| 9691031c-5fbe-437f-ae6d-90e7ae550175 | server-0x7f883335d350-0 | ACTIVE | - | Running | private=10.0.0.7, 172.24.4.3 |
| 204f51ed-3baa-4c50-9227-391a7c5f7c47 | server-0x7f8833395890-0 | ACTIVE | - | Running | private=10.0.0.2 |
| 353dfff7-7ce2-4b0d-8cc1-111510c0008b | server-0x7f88333a7ad0-0 | ACTIVE | - | Running | private=10.0.0.4 |
| a062aa43-11ae-4ea3-9c35-3ed3766c1fdf | server-0x7f88333a7f50-0 | ACTIVE | - | Running | private=10.0.0.6, 172.24.4.6, 172.24.4.15 |
| 3344bb31-6441-41b5-9056-28afc27133d3 | server-0x7f88333e8a90-0 | ACTIVE | - | Running | private=10.0.0.5 |
| 74780a52-a389-4d5e-8a32-eaacf4849a6f | server-0x7f8833c15e90-0 | ACTIVE | - | Running | private=10.0.0.8 |
| f0dd2112-e948-4844-93dd-592e2207454d | server-0x7f8833c39c50-0 | ERROR | - | NOSTATE | private=10.0.0.3, 172.24.4.2, 172.24.4.7, 172.24.4.11, 172.24.4.17 |
| d4545f39-6a5c-40e3-99f4-f72c22d56fc7 | server-0x7f8833c39c50-1 | ACTIVE | - | Running | private=10.0.0.9, 172.24.4.9, 172.24.4.14, 172.24.4.16 |
+--------------------------------------+-------------------------+--------+------------+-------------+--------------------------------------------------------------------+
pcrews@mahmachine:~/git/mahrepo$ nova floating-ip-list
+-------------+--------------------------------------+----------+--------+
| Ip | Server Id | Fixed Ip | Pool |
+-------------+--------------------------------------+----------+--------+
| 172.24.4.1 | - | - | public |
| 172.24.4.4 | - | - | public |
| 172.24.4.5 | - | - | public |
| 172.24.4.8 | - | - | public |
| 172.24.4.9 | d4545f39-6a5c-40e3-99f4-f72c22d56fc7 | 10.0.0.9 | public |
| 172.24.4.10 | - | - | public |
| 172.24.4.12 | - | - | public |
| 172.24.4.13 | - | - | public |
| 172.24.4.16 | d4545f39-6a5c-40e3-99f4-f72c22d56fc7 | 10.0.0.9 | public |
| 172.24.4.17 | f0dd2112-e948-4844-93dd-592e2207454d | 10.0.0.3 | public |
+-------------+---------...

[Numan Siddique (numansiddique)]

I am not able to reproduce this issue.
When I try to associate a second floating ip to an instance, I get the below error message.
I am using neutron. Are you using nova network ?

$ nova floating-ip-list
+------------+-----------+----------+--------+
| Ip | Server Id | Fixed Ip | Pool |
+------------+-----------+----------+--------+
| 172.24.4.5 | - | 10.0.0.2 | public |
| 172.24.4.6 | - | - | public |
+------------+-----------+----------+--------+
[vagrant@vagrant-fedora20 devstack]$ nova list
+--------------------------------------+------+--------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------------------+
| 70e4210b-bde5-44dc-82e5-89de4b8c83a4 | test | ACTIVE | - | Running | private=10.0.0.2, 172.24.4.5 |
+--------------------------------------+------+--------+------------+-------------+------------------------------+
[vagrant@vagrant-fedora20 devstack]$ nova floating-ip-associate test 172.24.4.6
ERROR (BadRequest): Unable to associate floating ip 172.24.4.6 to fixed ip 10.0.0.2 for instance 70e4210b-bde5-44dc-82e5-89de4b8c83a4. Error: Cannot associate floating IP 172.24.4.6 (ea005c03-9558-4e0e-9b00-200a35085ba6) with port fd0040ce-0118-4cdb-9213-ece2337a44d0 using fixed IP 10.0.0.2, as that fixed IP already has a floating IP on external network d448f95a-7903-4551-a348-66bb885b1f62. (HTTP 400) (Request-ID: req-893c15b9-b5f1-4c46-8833-048fb15a3b09)

[Numan Siddique (numansiddique)]

My test is on the latest master

[Patrick Crews (patrick-crews)]

This is with nova network and it is not an issue of associating a second floating ip with an instance, just that some vips can get into a state where they are attached, but unowned by the instance user via the sequence I describe above.

The example I give is of an instance with multiple vips, but that was just for convenience. The instance with 172.24.4.3 attached is in a similar state and that is the sole vip. Apologies if this caused any confusion.

[Marian Horban (mhorban)]

Hi Patrick Crews,
I tried to reproduce this bug on the latest sources. I haven't used neutron, but I used nova-network.
And I could not reproduce this bug using CLI and using test script which uses HTTP API(I've attached it).

My steps using CLI were:
1. get list of servers:
$nova list
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------+
| 6e1c4587-07ce-4ca1-90f4-06dbe279f28c | test | ACTIVE | - | Running | private=10.0.0.2 |
| 8451c931-c967-4129-8b19-0863969dcb9a | test | ACTIVE | - | Running | private=10.0.0.3 |
| c51bdca3-4ba4-44e4-ba44-7a8284d3d880 | test | ACTIVE | - | Running | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------+
2. get floating IP list:
$nova floating-ip-list
+------------+-----------+----------+--------+
| Ip | Server Id | Fixed Ip | Pool |
+------------+-----------+----------+--------+
| 172.24.4.1 | - | - | public |
| 172.24.4.2 | - | - | public |
| 172.24.4.4 | - | - | public |
| 172.24.4.6 | - | - | public |
| 172.24.4.7 | - | - | public |
+------------+-----------+----------+--------+
3. delete floating IP:
$nova floating-ip-delete 172.24.4.4
4. check floating IP list:
$nova floating-ip-list
+------------+-----------+----------+--------+
| Ip | Server Id | Fixed Ip | Pool |
+------------+-----------+----------+--------+
| 172.24.4.1 | - | - | public |
| 172.24.4.2 | - | - | public |
| 172.24.4.6 | - | - | public |
| 172.24.4.7 | - | - | public |
+------------+-----------+----------+--------+
5. associate deleted floating IP with server:
$nova floating-ip-associate 6e1c4587-07ce-4ca1-90f4-06dbe279f28c 172.24.4.4
6. check servers list:
$nova list
+--------------------------------------+------+--------+------------+-------------+------------------------------+
| ID | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+------------------------------+
| 6e1c4587-07ce-4ca1-90f4-06dbe279f28c | test | ACTIVE | - | Running | private=10.0.0.2, 172.24.4.4 |
| 8451c931-c967-4129-8b19-0863969dcb9a | test | ACTIVE | - | Running | private=10.0.0.3 |
| c51bdca3-4ba4-44e4-ba44-7a8284d3d880 | test | ACTIVE | - | Running | private=10.0.0.4 |
+--------------------------------------+------+--------+------------+-------------+------------------------------+
7. disassociate deleted floating IP:
$nova floating-ip-disassociate 6e1c4587-07ce-4ca1-90f4-06dbe279f28c 172.24.4.4
8. check servers list:
+--------------------------------------+------+--------+------------+-------------+------------------+
| ID ...

[melanie witt (melwitt)]

Setting to high because race condition.

[Patrick Crews (patrick-crews)]

To repeat / view this:
1) git clone https://github.com/pcrews/rannsaka.git
2) from ransakka/ransakka, use this command line:
     python rannsaka.py --host=http://$KEYSTONE_IP_ADDR --requests 20000 -w 15 --test-file=locust_files/floating_ip_stress.py

NOTE: This will run as-is against a devstack installation (assuming standard location of config files, etc0

This will execute a sequence of test actions that trigger the bug / create orphan ip's on my test system.

[Davanum Srinivas (DIMS) (dims-v)]

Please switch this back to "In Progress" when there are active reviews.

[Davanum Srinivas (DIMS) (dims-v)]

Marian, Can you please confirm that this is the review for this bug?
https://review.openstack.org/#/c/149358/

thanks,
dims

[Marian Horban (mhorban)]

Davanum,
Right you are, review in progress
Regards, Marian