Address re-allocation before DHCP lease's expire

[Stephen Gran]

This is related to https://bugs.launchpad.net/bugs/1022804

Running a Folsom install on Ubuntu precise, I am seeing:

I'm getting log lines of the form:
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:f6:fa:8c
Feb 5 11:18:08 ucsprodcntl04 dnsmasq-dhcp[19754]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:6f:72:31 no address available

Which led me to debug further and find this sequence:

initial allocation:

2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:44:00:53
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.9.240
2013-02-05 14:20:31 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)

<VM killed>
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] Hold allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/18487533-f064-43ea-bdce-dcbae2b01968)
2013-02-05 14:21:42 DEBUG [quantum.db.db_base_plugin_v2] 10.252.8.55 (14afac28-dae4-438f-b18b-c999b731b1a7) is being held

<New VM created>
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: created new 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: last match for 10.252.8.55-10.252.8.55
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Recycle: updated last 10.252.8.55-10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Delete allocated IP 10.252.8.56 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7)
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Generated mac for network de934a28-d1d3-4921-8c63-f432fae8b08d is fa:16:3e:06:ee:05
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP - 10.252.8.55 from 10.252.8.55 to 10.252.8.56
2013-02-05 14:24:06 DEBUG [quantum.db.db_base_plugin_v2] Allocated IP 10.252.8.55 (de934a28-d1d3-4921-8c63-f432fae8b08d/14afac28-dae4-438f-b18b-c999b731b1a7/5e241fe4-0290-4f68-bdf3-537688e99a0e)

Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: not using configured address 10.252.8.55 because it is leased to fa:16:3e:44:00:53
Feb 5 14:25:49 ucsprodcntl04 dnsmasq-dhcp[6945]: DHCPDISCOVER(tap7a23bd1f-16) fa:16:3e:06:ee:05 no address available

Which I've finally tracked down to two issues:

1st) It turns out that dnsmasq's lease time is controlled with the setting 'dhcp_lease_time' in dhcp_agent.ini, while quantum-server's database entry for expiration time is controlled by the setting 'dhcp_lease_duration' in quantum.conf.

Can I ask why there are 2 flags for this? I can't see the utility of being able to purposefully configure them to do different things, and it make it easy for an admin to miss one. Can the functionality please be merged?

2nd) It seems that quantum/db/db_base_plugin_v2.py:update_fixed_ip_lease_expiration does not in fact update the database, allowing the IP address of VMs that have been running for some time to be prematurely recycled.

Please let me know what I can do to help.

[dan wendlandt (danwent)]

need someone from ipam/dhcp team to look into this. assigning to G-3 as we should triage by that point.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/21317

[Stephen Gran (sgran)]

There is a working patch on review, but it is languishing due to lack of interest. Would someone mind taking a look?

[Gary Kotton (garyk)]

On 02/13/2013 04:23 PM, Stephen Gran wrote:
> There is a working patch on review, but it is languishing due to lack of
> interest. Would someone mind taking a look?
>

I am looking at it and trying to reproduce.

[Stephen Gran (sgran)]

In order to reproduce, all I had to do was run postgres with query logging enabled, and watch as clients renewed their DHCP lease. If you need a hand setting up such an environment, let me know.

Cheers,

[Gary Kotton (garyk)]

I see the following:

2013-02-13 11:08:53.042 24723 DEBUG quantum.db.dhcp_rpc_base [-] Updating lease expiration for 10.0.0.14 on network b1545aa0-204c-47c5-b5f3-0e4af0213815 from ubuntu-devstack. update_lease_expiration /opt/stack/quantum/quantum/db/dhcp_rpc_base.py:166

I will check the DB...

[Gary Kotton (garyk)]

Please see the gerrit commits. Validated your fix and it works.

[Stephen Gran (sgran)]

I've validated your fix to my fix, and it works as well :) I've pushed a new patch.

Thanks,

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21317
Committed: http://github.com/openstack/quantum/commit/bfa9d88f432f20cf4a4d85a4050e4f94dc874457
Submitter: Jenkins
Branch: master

commit bfa9d88f432f20cf4a4d85a4050e4f94dc874457
Author: Stephen Gran <email address hidden>
Date: Wed Feb 6 14:57:57 2013 +0000

    Persist updated expiration time

    Without creating a subtransaction, the database query wasn't actually
    being run and the data was being lost. This resulted in the case that
    on termination of long running VMs, the VM IP address was immediately
    available for reuse instead of being held as it should have been.

    Fixes: bug #1116500

    Change-Id: I7774273b8e799d945c27329e9da2dba34e39fdb8
    Signed-off-by: Stephen Gran <email address hidden>

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/21949

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21949
Committed: http://github.com/openstack/quantum/commit/b14824fd525d7c21977532aad2befab3555a17eb
Submitter: Jenkins
Branch: stable/folsom

commit b14824fd525d7c21977532aad2befab3555a17eb
Author: Stephen Gran <email address hidden>
Date: Wed Feb 6 14:57:57 2013 +0000

    Persist updated expiration time

    Without creating a subtransaction, the database query wasn't actually
    being run and the data was being lost. This resulted in the case that
    on termination of long running VMs, the VM IP address was immediately
    available for reuse instead of being held as it should have been.

    Fixes: bug #1116500

    Change-Id: I7774273b8e799d945c27329e9da2dba34e39fdb8
    Signed-off-by: Stephen Gran <email address hidden>

[seth vidal (skvidal)]

So - This issue also occurs in nova in folsom. Is there a patch for folsom for nova that corrects this as well as for quantum?