Address re-allocation before DHCP lease's expire

Bug #1116500 reported by Stephen Gran
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Stephen Gran
Folsom
Fix Released
High
Gary Kotton

Bug Description

This is related to https://bugs.launchpad.net/bugs/1022804

Running a Folsom install on Ubuntu precise, I am seeing:

I'm getting log lines of the form:
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:f6:fa:8c
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:6f:72:31 no address available

Which led me to debug further and find this sequence:

initial allocation:

2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:44:00:53
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.9.240
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)

<VM killed>
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] Hold allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] 10.252.8.55 (14afac28-dae4-438f-b18b-c999b731b1a7) is being held

<New VM created>
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: created new 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: last match for 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: updated last 10.252.8.55-10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.56 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:06:ee:05
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/5e241fe4-0290-4f68-bdf3-537688e99a0e)

Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:44:00:53
Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:06:ee:05 no address available

Which I've finally tracked down to two issues:

1st) It turns out that dnsmasq's lease time is controlled with the setting 'dhcp_lease_time' in dhcp_agent.ini, while quantum-server's database entry for expiration time is controlled by the setting 'dhcp_lease_duration' in quantum.conf.

Can I ask why there are 2 flags for this? I can't see the utility of being able to purposefully configure them to do different things, and it make it easy for an admin to miss one. Can the functionality please be merged?

2nd) It seems that quantum/db/db_base_plugin_v2.py:update_fixed_ip_lease_expiration does not in fact update the database, allowing the IP address of VMs that have been running for some time to be prematurely recycled.

Please let me know what I can do to help.

Revision history for this message
dan wendlandt (danwent) wrote :

need someone from ipam/dhcp team to look into this. assigning to G-3 as we should triage by that point.

Changed in quantum:
milestone: none → grizzly-3
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (master)

Fix proposed to branch: master
Review: https://review.openstack.org/21317

Changed in quantum:
assignee: nobody → Stephen Gran (sgran)
status: New → In Progress
Gary Kotton (garyk)
tags: added: folsom-backport-potential
Revision history for this message
Stephen Gran (sgran) wrote :

There is a working patch on review, but it is languishing due to lack of interest. Would someone mind taking a look?

Revision history for this message
Gary Kotton (garyk) wrote : Re: [Bug 1116500] Re: Address re-allocation before DHCP lease's expire

On 02/13/2013 04:23 PM, Stephen Gran wrote:
> There is a working patch on review, but it is languishing due to lack of
> interest. Would someone mind taking a look?
>

I am looking at it and trying to reproduce.

Revision history for this message
Stephen Gran (sgran) wrote :

In order to reproduce, all I had to do was run postgres with query logging enabled, and watch as clients renewed their DHCP lease. If you need a hand setting up such an environment, let me know.

Cheers,

Revision history for this message
Gary Kotton (garyk) wrote :

I see the following:

2013-02-13 11:08:53.042 24723 DEBUG quantum.db.dhcp_rpc_base [-] Updating lease expiration for 10.0.0.14 on network b1545aa0-204c-47c5-b5f3-0e4af0213815 from ubuntu-devstack. update_lease_expiration /opt/stack/quantum/quantum/db/dhcp_rpc_base.py:166

I will check the DB...

Revision history for this message
Gary Kotton (garyk) wrote :

Please see the gerrit commits. Validated your fix and it works.

Revision history for this message
Stephen Gran (sgran) wrote :

I've validated your fix to my fix, and it works as well :) I've pushed a new patch.

Thanks,

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (master)

Reviewed: https://review.openstack.org/21317
Committed: http://github.com/openstack/quantum/commit/bfa9d88f432f20cf4a4d85a4050e4f94dc874457
Submitter: Jenkins
Branch: master

commit bfa9d88f432f20cf4a4d85a4050e4f94dc874457
Author: Stephen Gran <email address hidden>
Date: Wed Feb 6 14:57:57 2013 +0000

    Persist updated expiration time

    Without creating a subtransaction, the database query wasn't actually
    being run and the data was being lost. This resulted in the case that
    on termination of long running VMs, the VM IP address was immediately
    available for reuse instead of being held as it should have been.

    Fixes: bug #1116500

    Change-Id: I7774273b8e799d945c27329e9da2dba34e39fdb8
    Signed-off-by: Stephen Gran <email address hidden>

Changed in quantum:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to quantum (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/21949

Thierry Carrez (ttx)
Changed in quantum:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to quantum (stable/folsom)

Reviewed: https://review.openstack.org/21949
Committed: http://github.com/openstack/quantum/commit/b14824fd525d7c21977532aad2befab3555a17eb
Submitter: Jenkins
Branch: stable/folsom

commit b14824fd525d7c21977532aad2befab3555a17eb
Author: Stephen Gran <email address hidden>
Date: Wed Feb 6 14:57:57 2013 +0000

    Persist updated expiration time

    Without creating a subtransaction, the database query wasn't actually
    being run and the data was being lost. This resulted in the case that
    on termination of long running VMs, the VM IP address was immediately
    available for reuse instead of being held as it should have been.

    Fixes: bug #1116500

    Change-Id: I7774273b8e799d945c27329e9da2dba34e39fdb8
    Signed-off-by: Stephen Gran <email address hidden>

tags: added: in-stable-folsom
Thierry Carrez (ttx)
Changed in quantum:
milestone: grizzly-3 → 2013.1
Alan Pevec (apevec)
tags: removed: folsom-backport-potential in-stable-folsom
Revision history for this message
seth vidal (skvidal) wrote :

So - This issue also occurs in nova in folsom. Is there a patch for folsom for nova that corrects this as well as for quantum?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers