nova w/ FlatDHCP isn't fully deallocating fixed_ips

Bug #953708 reported by Dan Prince
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Vish Ishaya

Bug Description

Using the most recent Nova Essex builds on Fedora. It appears that Nova isn't fully deallocating fixed_ips:

After deleting an instance I'm seeing the following warnings in network.log:

2012-03-13 04:29:17 WARNING nova.network.manager [req-4c766b6c-7e2f-4ce1-a021-4df6b75aeb48 None None] IP |192.168.0.2| leased that isn't allocated

----

The following error from nova-manage (I fixed this display issue in other bug):

[root@nova1 ~]# nova-manage fixed list
network IP address hostname host
192.168.0.0/24 192.168.0.0 None None
192.168.0.0/24 192.168.0.1 None None
Command failed, please check log for more info
2012-03-13 04:34:43 CRITICAL nova [req-4f51bcdf-0f87-4e50-a521-56536e24fbe2 None None] 2L

----

Here is the fixed IPs table:

mysql> select * from fixed_ips where instance_id is not null;
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+----------------------+------+
| created_at | updated_at | deleted_at | deleted | id | address | network_id | instance_id | allocated | leased | reserved | virtual_interface_id | host |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+----------------------+------+
| 2012-03-13 04:10:37 | 2012-03-13 04:33:20 | NULL | 0 | 3 | 192.168.0.2 | 1 | 2 | 0 | 1 | 0 | NULL | NULL |
+---------------------+---------------------+------------+---------+----+-------------+------------+-------------+-----------+--------+----------+----------------------+------+
1 row in set (0.00 sec)

----

Nova will work fine until an IP gets recycled at which point things go downhill.

Revision history for this message
Vish Ishaya (vishvananda) wrote :

I'm assuming you can't use force_dhcp_release, but i'm curious why _disassociate_stale_fixed_ips isn't working?

Dan Prince (dan-prince)
Changed in nova:
assignee: nobody → Dan Prince (dan-prince)
status: New → In Progress
Revision history for this message
Dan Prince (dan-prince) wrote :

Okay. I'm also seeing these in the logs when this happens...

2012-03-13 15:52:15 WARNING nova.utils [-] /usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py:330: Warning: Truncated incorrect DOUBLE value: '(6L,)'

Looks like fixed_ip_disassociate_all_by_timeout in the sql API is the problem. Apparently model_query on FixedIp.id returns an array of tuples... which makes the subsequent query unhappy... and thus the IPs hang around.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/5277

Dan Prince (dan-prince)
Changed in nova:
importance: Undecided → High
Changed in nova:
milestone: none → essex-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/5277
Committed: http://github.com/openstack/nova/commit/f48f1bb72474e564f4f8b01da97730f612a868f9
Submitter: Jenkins
Branch: master

commit f48f1bb72474e564f4f8b01da97730f612a868f9
Author: Dan Prince <email address hidden>
Date: Tue Mar 13 13:15:49 2012 -0400

    Make fixed_ip_disassociate_all_by_timeout work.

    Fix an issue in the SQL api where fixed_ip_disassociate_all_by_timeout
    was getting an array of tuples instead of IDs from a model_query.

    This should resolves issues where fixed IPs don't get dissassociated
    via the network managers periodic task.

    Fixes LP Bug #953708.

    Change-Id: Iac485cdcda072ee248237e6c4a31113685f5b980

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/5872

Changed in nova:
assignee: Dan Prince (dan-prince) → justinsb (justin-fathomdb)
status: Fix Released → In Progress
Thierry Carrez (ttx)
Changed in nova:
assignee: justinsb (justin-fathomdb) → nobody
milestone: essex-rc1 → none
status: In Progress → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: none → 2012.1
Changed in nova:
assignee: nobody → Vish Ishaya (vishvananda)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.