fixed_ips cannot reliably be released on instance termination

Bug #1103260 reported by Chet Burgess
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
High
Chet Burgess
Folsom
Fix Released
High
Vish Ishaya

Bug Description

On rare occasion we have noticed that using nova-network and the VlanManage fixed_ip aren't always marked as free during instance termination until the _disassociate_stale_fixed_ips periodic job is run.

The issue occurs as a result of dnsmasq is restarted. On restart the nova-dhcpbridge command is called with the init arguement to populate the dnsmasq lease DB. The lease time, is presently being calculated by taking instances.updated_at or instances.created_at and adding CONF.dhcp_lease_time. Unfortunately these fields are not updated by normal DHCP actions. At dnsmasq start time the old leases are loaded and immediately expired. As a result of this nova-dhcpbridge is called with the del action and fixed_ips.leased is marked as 0 in the DB even though a VM has a valid lease.

If a termination is issued while in this state dnsmasq will receive the release packet and drop it claiming the lease is unknown. As a result nova-dhcpbridge is not called to free the lease in the DB and the IP remains in a partially freed state until the _disassociate_stale_fixed_ips periodic job is run.

During normal DHCP operations the fixed_ips.updated_at field is updated on a lease request or renewal. As such the dnsmasq lease DB should be initialized from fixed_ips.updated_at or fixed_ips.created_at instead of the values from the instances table.

Chet Burgess (cfb-n)
Changed in nova:
assignee: nobody → Chet Burgess (cfb-n)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/20365

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/20453

Revision history for this message
Chet Burgess (cfb-n) wrote :

nova-dhcpbridge does not currently update fixed_ips.udpated_at when a lease is renenwed. This was done to reduce load on the nova DB. After conversations with nova community it was decided to not put additional load on the nova DB at this time. Instead will use the current UTC time instead.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/20453
Committed: http://github.com/openstack/nova/commit/e12339def3d9182f5ab23869e6bdc946e8e42825
Submitter: Jenkins
Branch: master

commit e12339def3d9182f5ab23869e6bdc946e8e42825
Author: Chet Burgess <email address hidden>
Date: Thu Jan 24 03:18:40 2013 +0000

    populate dnsmasq lease db with valid leases

    At start dnsmasq retrieves its lease db by calling the nova-dhcpbridge
    script with the init action.

    Previously we were returning all allocated IPs in the network and
    calculating the lease exirpation time based upon instances.updated_at
    or instances.created_at.

    Now we are only returning entries for IPs that are allocated and leased.
    Additionally we set the lease expiration time to be i
    now + CONF.dhcp_lease_time

    Change-Id: Ie3b4fafd38094dc01ceea592ab5229e581193512
    Fixes: bug #1103260

Changed in nova:
status: In Progress → Fix Committed
Changed in nova:
importance: Undecided → High
tags: added: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/folsom)

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/21057

Thierry Carrez (ttx)
Changed in nova:
milestone: none → grizzly-3
status: Fix Committed → Fix Released
Mark McLoughlin (markmc)
tags: removed: folsom-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/folsom)

Reviewed: https://review.openstack.org/21057
Committed: http://github.com/openstack/nova/commit/53626bf46d7730d026b36bcbefcac87e77028a6e
Submitter: Jenkins
Branch: stable/folsom

commit 53626bf46d7730d026b36bcbefcac87e77028a6e
Author: Chet Burgess <email address hidden>
Date: Thu Jan 24 03:18:40 2013 +0000

    populate dnsmasq lease db with valid leases

    At start dnsmasq retrieves its lease db by calling the nova-dhcpbridge
    script with the init action.

    Previously we were returning all allocated IPs in the network and
    calculating the lease exirpation time based upon instances.updated_at
    or instances.created_at.

    Now we are only returning entries for IPs that are allocated and leased.
    Additionally we set the lease expiration time to be i
    now + CONF.dhcp_lease_time

    Change-Id: Ie3b4fafd38094dc01ceea592ab5229e581193512
    Fixes: bug #1103260
    (cherry picked from commit e12339def3d9182f5ab23869e6bdc946e8e42825)

Thierry Carrez (ttx)
Changed in nova:
milestone: grizzly-3 → 2013.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.