fixed_ips cannot reliably be released on instance termination

[Chet Burgess]

On rare occasion we have noticed that using nova-network and the VlanManage fixed_ip aren't always marked as free during instance termination until the _disassociate_stale_fixed_ips periodic job is run.

The issue occurs as a result of dnsmasq is restarted. On restart the nova-dhcpbridge command is called with the init arguement to populate the dnsmasq lease DB. The lease time, is presently being calculated by taking instances.updated_at or instances.created_at and adding CONF.dhcp_lease_time. Unfortunately these fields are not updated by normal DHCP actions. At dnsmasq start time the old leases are loaded and immediately expired. As a result of this nova-dhcpbridge is called with the del action and fixed_ips.leased is marked as 0 in the DB even though a VM has a valid lease.

If a termination is issued while in this state dnsmasq will receive the release packet and drop it claiming the lease is unknown. As a result nova-dhcpbridge is not called to free the lease in the DB and the IP remains in a partially freed state until the _disassociate_stale_fixed_ips periodic job is run.

During normal DHCP operations the fixed_ips.updated_at field is updated on a lease request or renewal. As such the dnsmasq lease DB should be initialized from fixed_ips.updated_at or fixed_ips.created_at instead of the values from the instances table.

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/20365

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: master
Review: https://review.openstack.org/20453

[Chet Burgess (cfb-n)]

nova-dhcpbridge does not currently update fixed_ips.udpated_at when a lease is renenwed. This was done to reduce load on the nova DB. After conversations with nova community it was decided to not put additional load on the nova DB at this time. Instead will use the current UTC time instead.

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/20453
Committed: http://github.com/openstack/nova/commit/e12339def3d9182f5ab23869e6bdc946e8e42825
Submitter: Jenkins
Branch: master

commit e12339def3d9182f5ab23869e6bdc946e8e42825
Author: Chet Burgess <email address hidden>
Date: Thu Jan 24 03:18:40 2013 +0000

    populate dnsmasq lease db with valid leases

    At start dnsmasq retrieves its lease db by calling the nova-dhcpbridge
    script with the init action.

    Previously we were returning all allocated IPs in the network and
    calculating the lease exirpation time based upon instances.updated_at
    or instances.created_at.

    Now we are only returning entries for IPs that are allocated and leased.
    Additionally we set the lease expiration time to be i
    now + CONF.dhcp_lease_time

    Change-Id: Ie3b4fafd38094dc01ceea592ab5229e581193512
    Fixes: bug #1103260

[OpenStack Infra (hudson-openstack)]

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/21057

[OpenStack Infra (hudson-openstack)]

Reviewed: https://review.openstack.org/21057
Committed: http://github.com/openstack/nova/commit/53626bf46d7730d026b36bcbefcac87e77028a6e
Submitter: Jenkins
Branch: stable/folsom

commit 53626bf46d7730d026b36bcbefcac87e77028a6e
Author: Chet Burgess <email address hidden>
Date: Thu Jan 24 03:18:40 2013 +0000

    populate dnsmasq lease db with valid leases

    At start dnsmasq retrieves its lease db by calling the nova-dhcpbridge
    script with the init action.

    Previously we were returning all allocated IPs in the network and
    calculating the lease exirpation time based upon instances.updated_at
    or instances.created_at.

    Now we are only returning entries for IPs that are allocated and leased.
    Additionally we set the lease expiration time to be i
    now + CONF.dhcp_lease_time

    Change-Id: Ie3b4fafd38094dc01ceea592ab5229e581193512
    Fixes: bug #1103260
    (cherry picked from commit e12339def3d9182f5ab23869e6bdc946e8e42825)