Address re-allocation before DHCP lease's expire

[Mark McClain]

The current ip allocation scheme and DHCP agent do not coordinate on lease expiration. As a result, an address can be removed from a port and reassigned to a different port before the DHCP lease has expired.

[Mark McClain (markmcclain)]

We should be able to leverage RPC work in F3 to enable agent and the plugin to coordinate on lease expiration.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/11322

[dan wendlandt (danwent)]

this will be an FFE if it does not merge into F-3.

[Salvatore Orlando (salvatore-orlando)]

It's been approved a few hours ago.
Perhaps Jenkins needs a push?

[dan wendlandt (danwent)]

this review is stacked on top of another review (dhcp non-polling) which hasn't merged yet. If the changeset is independent of the non-polling stuff, we should unstack. Or we can just wait for the non-polling to merge.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/11322
Committed: http://github.com/openstack/quantum/commit/4ee0d1c54a92f258593ff3dbf2acc50075a5d8e1
Submitter: Jenkins
Branch: master

commit 4ee0d1c54a92f258593ff3dbf2acc50075a5d8e1
Author: Mark McClain <email address hidden>
Date: Mon Aug 13 23:18:54 2012 -0400

    Add model support for DHCP lease expiration

    bug 1022804

    This fix is phase 1 of a fix that to address re-allocation of a fixed_ip before
    the DHCP lease expires. The next phases will add RPC and recycling
    support.

    Change-Id: I3c907ade5cde562ff5a222bd97e56ffef367c722

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/11723

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/11723
Committed: http://github.com/openstack/quantum/commit/ed263e68e2236f5136df01373c4d66a433769e5f
Submitter: Jenkins
Branch: master

commit ed263e68e2236f5136df01373c4d66a433769e5f
Author: Mark McClain <email address hidden>
Date: Thu Aug 16 17:34:44 2012 -0400

    Add lease expiration script support for dnsmasq

    Fixes bug 1022804

    This is phase 2 of the bug fix. This changeset adds support for dnsmasq
     --dhcp-script to notify Quantum of lease renewals. Communication between
    dnsmasq and the Quantum DHCP agent occurs via UNIX domain socket since dnsmasq
    may run in a network namespace. The DHCP agent is responsible for
    relaying the updated lease expiration back the Quantum server.

    Change-Id: If42b76bbb9ec7543e681e26b9add8eb1d7054eeb

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/12227

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/12227
Committed: http://github.com/openstack/quantum/commit/dde6922b98c993ce4de355b9856d54b47197cd5d
Submitter: Jenkins
Branch: master

commit dde6922b98c993ce4de355b9856d54b47197cd5d
Author: Mark McClain <email address hidden>
Date: Wed Aug 29 13:56:50 2012 -0400

    Add lease expiration management to ip recycling

    Fixes bug 1022804

    This is the 3rd and final patch for this bug. This patch alters ip allocation
    recycling to honor lease expiration. Allocations that are in the
    expiration wait state have null port_ids.

    Change-Id: Ib7960b142eb15733c6418b01973d02a827634cb6

[Stephen Gran (sgran)]

I'm afraid I'm still seeing this issue on folsom. We have the lease time set to 3600 seconds, which is probably helping us trigger the issue.

I'm getting log lines of the form:
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:f6:fa:8c
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:6f:72:31 no address available

Please let me know what I can do to assist.

[Stephen Gran (sgran)]

seeing this sequence:

initial allocation:

2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:44:00:53
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.9.240
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)

<VM killed>
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] Hold allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] 10.252.8.55 (14afac28-dae4-438f-b18b-c999b731b1a7) is being held

<New VM created>
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: created new 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: last match for 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: updated last 10.252.8.55-10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.56 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:06:ee:05
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/5e241fe4-0290-4f68-bdf3-537688e99a0e)

Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:44:00:53
Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:06:ee:05 no address available

Again, let me know if I can help.

Cheers,

[Stephen Gran (sgran)]

OK, I finally tracked it down with the help of some printf debugging. It turns out that dnsmasq's lease time is controlled with the setting 'dhcp_lease_time' in dhcp_agent.ini, while quantum-server's database entry for expiration time is controlled by the setting 'dhcp_lease_duration' in quantum.conf.

Can I ask why there are 2 flags for this? I can't see the utility of being able to purposefully configure them to do different things, and it make it easy for an admin to miss one. Can the functionality please be merged?

Cheers,

[Mark McClain (markmcclain)]

Stephen-

Thanks for the bug report. Would you mind copying this information into a new bug? This bug was about adding the protection and it tied to an older release. I need to dig a little further, but it appears you've found a possible new bug that needs to targeted to later releases.

[Stephen Gran (sgran)]

OK, have done so: https://bugs.launchpad.net/quantum/+bug/1116500